Mailing List Archive

I've just had an unsuccessful attempt to upgrade to systemd-230-r1. It
segfaults and slows the system down. The symptoms are better compared to
-229, but still significant.

https://forums.grsecurity.net/viewtopic.php?f=3&t=4485

Some relevant log entries:
grsec: denied resource overstep by requesting 8392704 for RLIMIT_STACK
against limit 8388608 for /usr/lib64/systemd/systemd[systemd:2735]
uid/euid:0/0 gid/egid:0/0, parent /usr/lib64/systemd/systemd[systemd:1]
uid/euid:0/0 gid/egid:0/0
systemd[2735]: segfault at 39f8d01cf00 ip 00000368d4caa2e4 sp
0000039f8d01cf00 error 6 in libc-2.23.so[368d4c62000+19a000]
grsec: Segmentation fault occurred at 0000039f8d01cf00 in
/usr/lib64/systemd/systemd[systemd:2735] uid/euid:0/0 gid/egid:0/0, parent
/usr/lib64/systemd/systemd[systemd:1] uid/euid:0/0 gid/egid:0/0
grsec: bruteforce prevention initiated for the next 30 minutes or until
service restarted, stalling each fork 30 seconds. Please investigate the
crash report for /usr/lib64/systemd/systemd[systemd:2735] uid/euid:0/0
gid/egid:0/0, parent /usr/lib64/systemd/systemd[systemd:1] uid/euid:0/0
gid/egid:0/0

systemd-coredump[2747]: Process 2735 (systemd) of user 0 dumped core.

Stack trace of thread
2735:
#0 0x00000368d4caa2e4
_IO_vfprintf
(libc.so.6)
#1 0x00000368d4d5e852
__vsnprintf_chk
(libc.so.6)
#2 0x00000368d4d5e7a4
__snprintf_chk
(libc.so.6)
#3 0x00000000df8db344
n/a (systemd)
#4 0x00000000df8db9aa
n/a (systemd)
#5 0x00000000df8da72f
n/a (systemd)
#6 0x00000000df8db314
n/a (systemd)
#7 0x00000000df8db9aa
n/a (systemd)
#8 0x00000000df8da72f
n/a (systemd)
#9 0x00000000df8db314
n/a (systemd)
#10 0x00000000df8db9aa
n/a (systemd)
#11 0x00000000df8da72f
n/a (systemd)
#12 0x00000000df8db314
n/a (systemd)
#13 0x00000000df8db9aa
n/a (systemd)
#14 0x00000000df8da72f
n/a (systemd)
#15 0x00000000df8db314
n/a (systemd)
#16 0x00000000df8db9aa
n/a (systemd)
#17 0x00000000df8da72f
n/a (systemd)
#18 0x00000000df8db314
n/a (systemd)
#19 0x00000000df8db9aa
n/a (systemd)
#20 0x00000000df8da72f
n/a (systemd)
#21 0x00000000df8db314
n/a (systemd)
#22 0x00000000df8db9aa
n/a (systemd)
#23 0x00000000df8da72f
n/a (systemd)
#24 0x00000000df8db314
n/a (systemd)
#25 0x00000000df8db9aa
n/a (systemd)
#26 0x00000000df8da72f
n/a (systemd)
#27 0x00000000df8db314
n/a (systemd)
#28 0x00000000df8db9aa
n/a (systemd)
#29 0x00000000df8da72f
n/a (systemd)
#30 0x00000000df8db314
n/a (systemd)
#31 0x00000000df8db9aa
n/a (systemd)
#32 0x00000000df8da72f
n/a (systemd)
#33 0x00000000df8db314
n/a (systemd)
#34 0x00000000df8db9aa
n/a (systemd)
#35 0x00000000df8da72f
n/a (systemd)
#36 0x00000000df8db314
n/a (systemd)
#37 0x00000000df8db9aa
n/a (systemd)
#38 0x00000000df8da72f
n/a (systemd)
#39 0x00000000df8db314
n/a (systemd)
#40 0x00000000df8db9aa
n/a (systemd)
#41 0x00000000df8da72f
n/a (systemd)
#42 0x00000000df8db314
n/a (systemd)
#43 0x00000000df8db9aa
n/a (systemd)
#44 0x00000000df8da72f
n/a (systemd)
#45 0x00000000df8db314
n/a (systemd)
#46 0x00000000df8db9aa
n/a (systemd)
#47 0x00000000df8da72f
n/a (systemd)
#48 0x00000000df8db314
n/a (systemd)
#49 0x00000000df8db9aa
n/a (systemd)
#50 0x00000000df8da72f
n/a (systemd)
#51 0x00000000df8db314
n/a (systemd)
#52 0x00000000df8db9aa
n/a (systemd)
#53 0x00000000df8da72f
n/a (systemd)
#54 0x00000000df8db314
n/a (systemd)
#55 0x00000000df8db9aa
n/a (systemd)
#56 0x00000000df8da72f
n/a (systemd)
#57 0x00000000df8db314
n/a (systemd)
#58 0x00000000df8db9aa
n/a (systemd)
#59 0x00000000df8da72f
n/a (systemd)
#60 0x00000000df8db314
n/a (systemd)
#61 0x00000000df8db9aa
n/a (systemd)
#62 0x00000000df8da72f
n/a (systemd)
#63 0x00000000df8db314
n/a (systemd)
systemd-logind[897]: Failed to abandon session scope: Connection timed out


Any of you have problems with the latest versions of systemd as well? Any
ideas?

Thanks:
Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

2016.Március 10.(Cs) 01:53 időpontban "Tóth Attila" ezt írta:
> After upgrading to systemd-229 it segfaults early during boot triggering
> bruteforce prevention, which renders the system annoyingly slow.
>
> grsec: Segmentation fault occurred at 000003e45975efd0 in
> /usr/lib64/systemd/systemd[systemd:1135]
> grsec: bruteforce prevention initiated for the next 30 minutes or until
> service restarted, stalling each fork 30 seconds. Please investigate the
> crash report for /usr/lib64/systemd/systemd[systemd:1135]
>
> Avoid it or be aware that might happen: Dw.
> --
> dr Tóth Attila, Radiológus, 06-20-825-8057
> Attila Toth MD, Radiologist, +36-20-825-8057
>
>
>
>

By looking at the addresses in the stack trace, is it me or is it a
case of a stack overflow beacuse of an infinite recursion?

Lennart gave another reason to stay away from his code.

--
René Rhéaume

On Wed, Jun 1, 2016, at 15:49, Tóth Attila wrote:
> I've just had an unsuccessful attempt to upgrade to systemd-230-r1. It
> segfaults and slows the system down. The symptoms are better compared to
> -229, but still significant.
>
> https://forums.grsecurity.net/viewtopic.php?f=3&t=4485
>
> Some relevant log entries:
> grsec: denied resource overstep by requesting 8392704 for RLIMIT_STACK
> against limit 8388608 for /usr/lib64/systemd/systemd[systemd:2735]
> uid/euid:0/0 gid/egid:0/0, parent /usr/lib64/systemd/systemd[systemd:1]
> uid/euid:0/0 gid/egid:0/0
> systemd[2735]: segfault at 39f8d01cf00 ip 00000368d4caa2e4 sp
> 0000039f8d01cf00 error 6 in libc-2.23.so[368d4c62000+19a000]
> grsec: Segmentation fault occurred at 0000039f8d01cf00 in
> /usr/lib64/systemd/systemd[systemd:2735] uid/euid:0/0 gid/egid:0/0,
> parent
> /usr/lib64/systemd/systemd[systemd:1] uid/euid:0/0 gid/egid:0/0
> grsec: bruteforce prevention initiated for the next 30 minutes or until
> service restarted, stalling each fork 30 seconds. Please investigate the
> crash report for /usr/lib64/systemd/systemd[systemd:2735] uid/euid:0/0
> gid/egid:0/0, parent /usr/lib64/systemd/systemd[systemd:1] uid/euid:0/0
> gid/egid:0/0
>
> systemd-coredump[2747]: Process 2735 (systemd) of user 0 dumped core.
>
> Stack trace of thread
> 2735:
> #0 0x00000368d4caa2e4
> _IO_vfprintf
> (libc.so.6)
> #1 0x00000368d4d5e852
> __vsnprintf_chk
> (libc.so.6)
> #2 0x00000368d4d5e7a4
> __snprintf_chk
> (libc.so.6)
> #3 0x00000000df8db344
> n/a (systemd)
> #4 0x00000000df8db9aa
> n/a (systemd)
> #5 0x00000000df8da72f
> n/a (systemd)
> #6 0x00000000df8db314
> n/a (systemd)
> #7 0x00000000df8db9aa
> n/a (systemd)
> #8 0x00000000df8da72f
> n/a (systemd)
> #9 0x00000000df8db314
> n/a (systemd)
> #10 0x00000000df8db9aa
> n/a (systemd)
> #11 0x00000000df8da72f
> n/a (systemd)
> #12 0x00000000df8db314
> n/a (systemd)
> #13 0x00000000df8db9aa
> n/a (systemd)
> #14 0x00000000df8da72f
> n/a (systemd)
> #15 0x00000000df8db314
> n/a (systemd)
> #16 0x00000000df8db9aa
> n/a (systemd)
> #17 0x00000000df8da72f
> n/a (systemd)
> #18 0x00000000df8db314
> n/a (systemd)
> #19 0x00000000df8db9aa
> n/a (systemd)
> #20 0x00000000df8da72f
> n/a (systemd)
> #21 0x00000000df8db314
> n/a (systemd)
> #22 0x00000000df8db9aa
> n/a (systemd)
> #23 0x00000000df8da72f
> n/a (systemd)
> #24 0x00000000df8db314
> n/a (systemd)
> #25 0x00000000df8db9aa
> n/a (systemd)
> #26 0x00000000df8da72f
> n/a (systemd)
> #27 0x00000000df8db314
> n/a (systemd)
> #28 0x00000000df8db9aa
> n/a (systemd)
> #29 0x00000000df8da72f
> n/a (systemd)
> #30 0x00000000df8db314
> n/a (systemd)
> #31 0x00000000df8db9aa
> n/a (systemd)
> #32 0x00000000df8da72f
> n/a (systemd)
> #33 0x00000000df8db314
> n/a (systemd)
> #34 0x00000000df8db9aa
> n/a (systemd)
> #35 0x00000000df8da72f
> n/a (systemd)
> #36 0x00000000df8db314
> n/a (systemd)
> #37 0x00000000df8db9aa
> n/a (systemd)
> #38 0x00000000df8da72f
> n/a (systemd)
> #39 0x00000000df8db314
> n/a (systemd)
> #40 0x00000000df8db9aa
> n/a (systemd)
> #41 0x00000000df8da72f
> n/a (systemd)
> #42 0x00000000df8db314
> n/a (systemd)
> #43 0x00000000df8db9aa
> n/a (systemd)
> #44 0x00000000df8da72f
> n/a (systemd)
> #45 0x00000000df8db314
> n/a (systemd)
> #46 0x00000000df8db9aa
> n/a (systemd)
> #47 0x00000000df8da72f
> n/a (systemd)
> #48 0x00000000df8db314
> n/a (systemd)
> #49 0x00000000df8db9aa
> n/a (systemd)
> #50 0x00000000df8da72f
> n/a (systemd)
> #51 0x00000000df8db314
> n/a (systemd)
> #52 0x00000000df8db9aa
> n/a (systemd)
> #53 0x00000000df8da72f
> n/a (systemd)
> #54 0x00000000df8db314
> n/a (systemd)
> #55 0x00000000df8db9aa
> n/a (systemd)
> #56 0x00000000df8da72f
> n/a (systemd)
> #57 0x00000000df8db314
> n/a (systemd)
> #58 0x00000000df8db9aa
> n/a (systemd)
> #59 0x00000000df8da72f
> n/a (systemd)
> #60 0x00000000df8db314
> n/a (systemd)
> #61 0x00000000df8db9aa
> n/a (systemd)
> #62 0x00000000df8da72f
> n/a (systemd)
> #63 0x00000000df8db314
> n/a (systemd)
> systemd-logind[897]: Failed to abandon session scope: Connection timed
> out
>
>
> Any of you have problems with the latest versions of systemd as well? Any
> ideas?
>
> Thanks:
> Dw.
> --
> dr Tóth Attila, Radiológus, 06-20-825-8057
> Attila Toth MD, Radiologist, +36-20-825-8057
>
> 2016.Március 10.(Cs) 01:53 időpontban "Tóth Attila" ezt írta:
> > After upgrading to systemd-229 it segfaults early during boot triggering
> > bruteforce prevention, which renders the system annoyingly slow.
> >
> > grsec: Segmentation fault occurred at 000003e45975efd0 in
> > /usr/lib64/systemd/systemd[systemd:1135]
> > grsec: bruteforce prevention initiated for the next 30 minutes or until
> > service restarted, stalling each fork 30 seconds. Please investigate the
> > crash report for /usr/lib64/systemd/systemd[systemd:1135]
> >
> > Avoid it or be aware that might happen: Dw.
> > --
> > dr Tóth Attila, Radiológus, 06-20-825-8057
> > Attila Toth MD, Radiologist, +36-20-825-8057

Not necessarily the ideal solution, but have you tried twiddling with
the stack size in limits.conf?

If I read this right, grsec limits the size of the stack, which causes
the process to segfault.

--
0x7D964D3361142ACF

2016.Június 2.(Cs) 02:31 időpontban Max R.D. Parmer ezt írta:
> On Wed, Jun 1, 2016, at 15:49, Tóth Attila wrote:
>> I've just had an unsuccessful attempt to upgrade to systemd-230-r1. It
>> segfaults and slows the system down. The symptoms are better compared to
>> -229, but still significant.
>>
>> https://forums.grsecurity.net/viewtopic.php?f=3&t=4485
>>
>> Some relevant log entries:
>> grsec: denied resource overstep by requesting 8392704 for RLIMIT_STACK
>> against limit 8388608 for /usr/lib64/systemd/systemd[systemd:2735]
>> uid/euid:0/0 gid/egid:0/0, parent /usr/lib64/systemd/systemd[systemd:1]
>> uid/euid:0/0 gid/egid:0/0
>> systemd[2735]: segfault at 39f8d01cf00 ip 00000368d4caa2e4 sp
>> 0000039f8d01cf00 error 6 in libc-2.23.so[368d4c62000+19a000]
>> grsec: Segmentation fault occurred at 0000039f8d01cf00 in
>> /usr/lib64/systemd/systemd[systemd:2735] uid/euid:0/0 gid/egid:0/0,
>> parent
>> /usr/lib64/systemd/systemd[systemd:1] uid/euid:0/0 gid/egid:0/0
>> grsec: bruteforce prevention initiated for the next 30 minutes or until
>> service restarted, stalling each fork 30 seconds. Please investigate
>> the
>> crash report for /usr/lib64/systemd/systemd[systemd:2735] uid/euid:0/0
>> gid/egid:0/0, parent /usr/lib64/systemd/systemd[systemd:1] uid/euid:0/0
>> gid/egid:0/0
>>
>> systemd-coredump[2747]: Process 2735 (systemd) of user 0 dumped core.
>>
>> Stack trace of thread
>> 2735:
>> #0
>> 0x00000368d4caa2e4
>> _IO_vfprintf
>> (libc.so.6)
>> #1
>> 0x00000368d4d5e852
>> __vsnprintf_chk
>> (libc.so.6)
>> #2
>> 0x00000368d4d5e7a4
>> __snprintf_chk
>> (libc.so.6)
>> #3
>> 0x00000000df8db344
>> n/a (systemd)
>> #4
>> 0x00000000df8db9aa
>> n/a (systemd)
>
> Not necessarily the ideal solution, but have you tried twiddling with
> the stack size in limits.conf?

I checked an the system-wide defaults apply to systemd, which is: 8M soft
limit and _unlimited_ hard limit for stack size. So after exceeding soft
limit systemd segfaults and tries to dump core.

cat /proc/1/limits
Limit Soft Limit Hard Limit Units
Max stack size 8388608 unlimited bytes

I expect any process to handle a situation of trying to exceed soft limit
with unlimited hard limit in another way than segfaulting and attempting
to dump core...

> If I read this right, grsec limits the size of the stack, which causes
> the process to segfault.

I think grsec does not enforce any stack limits here, it just reports the
issue and makes it more visible.

BR: Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057

2016.Június 2.(Cs) 21:39 időpontban "Tóth Attila" ezt írta:
> 2016.Június 2.(Cs) 02:31 időpontban Max R.D. Parmer ezt írta:
>> Not necessarily the ideal solution, but have you tried twiddling with
>> the stack size in limits.conf?
>
> I checked an the system-wide defaults apply to systemd, which is: 8M soft
> limit and _unlimited_ hard limit for stack size. So after exceeding soft
> limit systemd segfaults and tries to dump core.
>
> cat /proc/1/limits
> Limit Soft Limit Hard Limit Units
> Max stack size 8388608 unlimited bytes
>
> I expect any process to handle a situation of trying to exceed soft limit
> with unlimited hard limit in another way than segfaulting and attempting
> to dump core...

Increasing the limit doesn't fix the issue - I'm not surprised about that.

For those who are not familiar: systemd doesn't respect limits.conf. In
system.conf the default values can be configured and per unit limits can
be specified. To my surprise, systemd doesn't seem to pay attention to
it's own configuration file. In order to provide increased stack limit for
init, I also modified the kernel defaults. With no success.

>> If I read this right, grsec limits the size of the stack, which causes
>> the process to segfault.
>
> I think grsec does not enforce any stack limits here, it just reports the
> issue and makes it more visible.

I did a bisect and it turns out a this commit is responsible for the
symptoms:
https://github.com/systemd/systemd/commit/d054f0a4d451120c26494263fc4dc175bfd405b1
tree-wide: use xsprintf() where applicable

I try to contact the developer. Whether he has an idea on what is
happening here.

BR: Dw.
--
dr Tóth Attila, Radiológus, 06-20-825-8057
Attila Toth MD, Radiologist, +36-20-825-8057