https://bz.apache.org/bugzilla/show_bug.cgi?id=67504
Bug ID: 67504
Summary: mod_fcgid can segfault at high memory utilization
(Cannot allocate memory)
Product: Apache httpd-2
Version: 2.4.52
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_fcgid
Assignee: bugs@httpd.apache.org
Reporter: gknw@jzfs.pl
Target Milestone: ---
Created attachment 39067
--> https://bz.apache.org/bugzilla/attachment.cgi?id=39067&action=edit
Handle segmentation fault
Good afternoon!
I have a system running apache2 inside a runc container and with cgroups
limits. Sometimes when there is out of memory situation, the fcgid process
manager crashes with errors in the apache2 error.log:
[fcgid:warn] [pid 10:tid 140136058713856] (12)Cannot allocate memory: [client
85.115.164.244:0] mod_fcgid: can't write spawn command
[core:notice] [pid 1:tid 140136148075648] AH00051: child pid 9 exit signal
Segmentation fault (11), possible coredump in /tmp
[fcgid:error] [pid 1:tid 140136148075648] mod_fcgid: fcgid process manager
died, restarting the server
Normally, processes tree looks like:
/bin/init
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /var/www/cgi-bin/php-cgi7.4
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
After segmentation fault processes tree looks like:
/bin/init
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
\_ /var/www/cgi-bin/php-cgi7.4
and PHP process never exits and still consumes resources. Also the fcgid socket
opened by PHP is never closed.
I can reproduce these errors by 200 parallel requests to a PHP script like
this:
<?php
file_get_contents("/dev/urandom");
?>
and loop through something to RAM utilization, like this:
#!/usr/bin/perl
my $DATA = 'A' x 450000000;
sleep 6000;
Dropping PHP processes to init cat be reproduced by sending SIGSEGV or SIGKILL
to fcgid process manager, and it's normal. So I need to avoid segmentation
fault at mod_fcgid.
GDB shows next coredump backtrace at segmentation fault:
(gdb) backtrace
#0 __strrchr_sse2 () at ../sysdeps/x86_64/multiarch/../strrchr.S:32
#1 0x0000556c388f28b9 in ap_make_dirstr_parent ()
#2 0x00007fba66340698 in proc_spawn_process () from
/usr/lib/apache2/modules/mod_fcgid.so
#3 0x00007fba6633d7ef in pm_main () from /usr/lib/apache2/modules/mod_fcgid.so
#4 0x00007fba6633f2a9 in procmgr_post_config () from
/usr/lib/apache2/modules/mod_fcgid.so
#5 0x00007fba66338a2c in ?? () from /usr/lib/apache2/modules/mod_fcgid.so
#6 0x0000556c38915bb3 in ap_run_post_config ()
#7 0x0000556c388eefa3 in main ()
Message "can't write spawn command" logs at `procmgr_send_spawn_cmd` function
of mod_fcgid. And `procmgr_send_spawn_cmd` exits with `APR_SUCCESS` even if
`apr_file_write_full` returns error. Also at function `handle_request` missed
return value checking for `procmgr_send_spawn_cmd`. This results in NULL
dereference at invoking `ap_make_dirstr_parent` inside `proc_spawn_process`
because `wargv[0]` may be NULL.
I'm prepared patch to show where does the segmentation fault occur (see
attachments). With that patch segmentation fault is handled, and there are no
PHP processes dropped to init (pid 1). Instead of segmentation fault next
messages presents at error.log:
[fcgid:error] [pid 10:tid 140593983431808] mod_fcgid: wargv[0] is NULL. Handle
SIGSEGV
[fcgid:warn] [pid 10:tid 140593983431808] (20014)Internal error (specific
information not available): mod_fcgid: spawn process error
[fcgid:error] [pid 10:tid 140593983431808] mod_fcgid: wargv[0] is NULL. Handle
SIGSEGV
[fcgid:warn] [pid 10:tid 140593983431808] (20014)Internal error (specific
information not available): mod_fcgid: spawn process error
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
Bug ID: 67504
Summary: mod_fcgid can segfault at high memory utilization
(Cannot allocate memory)
Product: Apache httpd-2
Version: 2.4.52
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_fcgid
Assignee: bugs@httpd.apache.org
Reporter: gknw@jzfs.pl
Target Milestone: ---
Created attachment 39067
--> https://bz.apache.org/bugzilla/attachment.cgi?id=39067&action=edit
Handle segmentation fault
Good afternoon!
I have a system running apache2 inside a runc container and with cgroups
limits. Sometimes when there is out of memory situation, the fcgid process
manager crashes with errors in the apache2 error.log:
[fcgid:warn] [pid 10:tid 140136058713856] (12)Cannot allocate memory: [client
85.115.164.244:0] mod_fcgid: can't write spawn command
[core:notice] [pid 1:tid 140136148075648] AH00051: child pid 9 exit signal
Segmentation fault (11), possible coredump in /tmp
[fcgid:error] [pid 1:tid 140136148075648] mod_fcgid: fcgid process manager
died, restarting the server
Normally, processes tree looks like:
/bin/init
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /var/www/cgi-bin/php-cgi7.4
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
After segmentation fault processes tree looks like:
/bin/init
\_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
| \_ /usr/sbin/apache2 -T -DNO_DETACH -f
/etc/apache2/example.com/conf/apache2.conf
\_ /var/www/cgi-bin/php-cgi7.4
and PHP process never exits and still consumes resources. Also the fcgid socket
opened by PHP is never closed.
I can reproduce these errors by 200 parallel requests to a PHP script like
this:
<?php
file_get_contents("/dev/urandom");
?>
and loop through something to RAM utilization, like this:
#!/usr/bin/perl
my $DATA = 'A' x 450000000;
sleep 6000;
Dropping PHP processes to init cat be reproduced by sending SIGSEGV or SIGKILL
to fcgid process manager, and it's normal. So I need to avoid segmentation
fault at mod_fcgid.
GDB shows next coredump backtrace at segmentation fault:
(gdb) backtrace
#0 __strrchr_sse2 () at ../sysdeps/x86_64/multiarch/../strrchr.S:32
#1 0x0000556c388f28b9 in ap_make_dirstr_parent ()
#2 0x00007fba66340698 in proc_spawn_process () from
/usr/lib/apache2/modules/mod_fcgid.so
#3 0x00007fba6633d7ef in pm_main () from /usr/lib/apache2/modules/mod_fcgid.so
#4 0x00007fba6633f2a9 in procmgr_post_config () from
/usr/lib/apache2/modules/mod_fcgid.so
#5 0x00007fba66338a2c in ?? () from /usr/lib/apache2/modules/mod_fcgid.so
#6 0x0000556c38915bb3 in ap_run_post_config ()
#7 0x0000556c388eefa3 in main ()
Message "can't write spawn command" logs at `procmgr_send_spawn_cmd` function
of mod_fcgid. And `procmgr_send_spawn_cmd` exits with `APR_SUCCESS` even if
`apr_file_write_full` returns error. Also at function `handle_request` missed
return value checking for `procmgr_send_spawn_cmd`. This results in NULL
dereference at invoking `ap_make_dirstr_parent` inside `proc_spawn_process`
because `wargv[0]` may be NULL.
I'm prepared patch to show where does the segmentation fault occur (see
attachments). With that patch segmentation fault is handled, and there are no
PHP processes dropped to init (pid 1). Instead of segmentation fault next
messages presents at error.log:
[fcgid:error] [pid 10:tid 140593983431808] mod_fcgid: wargv[0] is NULL. Handle
SIGSEGV
[fcgid:warn] [pid 10:tid 140593983431808] (20014)Internal error (specific
information not available): mod_fcgid: spawn process error
[fcgid:error] [pid 10:tid 140593983431808] mod_fcgid: wargv[0] is NULL. Handle
SIGSEGV
[fcgid:warn] [pid 10:tid 140593983431808] (20014)Internal error (specific
information not available): mod_fcgid: spawn process error
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org