Mailing List Archive

httpd 2.4 CPU spike after graceful restart
Hi,

I noticed that sometimes when performing a graceful restart of httpd, the
CPU of the httpd process spikes and does not come down. This happens more
frequently if the web server is under load but can happen even when not
serving traffic. Has anyone else encountered this problem? And what can I
do to resolve this?

I did not notice any error messages in the error log and only see the
following:
Sat May 29 03:33:01.424748 2021] [mpm_worker:notice] [pid 2773:tid
140527163445824] AH00297: SIGUSR1 received. Doing graceful restart
AH00558: httpd: Could not reliably determine the server's fully qualified
domain name, using 127.0.0.1. Set the 'ServerName' directive globally to
suppress this message
[Sat May 29 03:33:01.485368 2021] [http2:warn] [pid 2773:tid
140527163445824] AH02951: mod_ssl does not seem to be enabled
[Sat May 29 03:33:01.485953 2021] [mpm_worker:notice] [pid 2773:tid
140527163445824] AH00292: Apache/2.4.46 () configured -- resuming normal
operations
[Sat May 29 03:33:01.485964 2021] [core:notice] [pid 2773:tid
140527163445824] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sat May 29 03:33:01.486045 2021] [mpm_worker:warn] [pid 2773:tid
140527163445824] AH00291: long lost child came home! (pid 25489)


From performing a top
top - 17:44:22 up 5 days, 16:04, 1 user, load average: 7.00, 7.13, 8.27
Threads: 384 total, 8 running, 316 sleeping, 0 stopped, 1 zombie
%Cpu(s): 99.8 us, 0.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem : 7807268 total, 4124256 free, 543316 used, 3139696 buff/cache
KiB Swap: 1310716 total, 1310716 free, 0 used. 6949164 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23677 apache 20 0 2293264 7524 4148 R 58.6 0.1 4582:47 httpd
23664 apache 20 0 2293264 7524 4148 R 58.2 0.1 4581:57 httpd
23636 apache 20 0 2293264 7524 4148 R 57.6 0.1 4582:16 httpd
23660 apache 20 0 2293264 7524 4148 R 57.2 0.1 4580:51 httpd
23698 apache 20 0 2293264 7524 4148 R 56.6 0.1 4582:17 httpd
23648 apache 20 0 2293264 7524 4148 R 55.9 0.1 4580:52 httpd
23668 apache 20 0 2293264 7524 4148 R 53.9 0.1 4581:23 httpd
1 root 20 0 128600 8592 5524 S 0.0 0.1 6:02.91 systemd

Regards
Jay
Re: httpd 2.4 CPU spike after graceful restart [ In reply to ]
On Thu, Jun 3, 2021 at 6:23 AM Jay Ce <jceh12021@gmail.com> wrote:
>
> Hi,
>
> I noticed that sometimes when performing a graceful restart of httpd, the CPU of the httpd process spikes and does not come down. This happens more frequently if the web server is under load but can happen even when not serving traffic. Has anyone else encountered this problem? And what can I do to resolve this?
>
> I did not notice any error messages in the error log and only see the following:
> Sat May 29 03:33:01.424748 2021] [mpm_worker:notice] [pid 2773:tid 140527163445824] AH00297: SIGUSR1 received. Doing graceful restart
> AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
> [Sat May 29 03:33:01.485368 2021] [http2:warn] [pid 2773:tid 140527163445824] AH02951: mod_ssl does not seem to be enabled
> [Sat May 29 03:33:01.485953 2021] [mpm_worker:notice] [pid 2773:tid 140527163445824] AH00292: Apache/2.4.46 () configured -- resuming normal operations
> [Sat May 29 03:33:01.485964 2021] [core:notice] [pid 2773:tid 140527163445824] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
> [Sat May 29 03:33:01.486045 2021] [mpm_worker:warn] [pid 2773:tid 140527163445824] AH00291: long lost child came home! (pid 25489)
>
>
> From performing a top
> top - 17:44:22 up 5 days, 16:04, 1 user, load average: 7.00, 7.13, 8.27
> Threads: 384 total, 8 running, 316 sleeping, 0 stopped, 1 zombie
> %Cpu(s): 99.8 us, 0.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> KiB Mem : 7807268 total, 4124256 free, 543316 used, 3139696 buff/cache
> KiB Swap: 1310716 total, 1310716 free, 0 used. 6949164 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 23677 apache 20 0 2293264 7524 4148 R 58.6 0.1 4582:47 httpd
> 23664 apache 20 0 2293264 7524 4148 R 58.2 0.1 4581:57 httpd
> 23636 apache 20 0 2293264 7524 4148 R 57.6 0.1 4582:16 httpd
> 23660 apache 20 0 2293264 7524 4148 R 57.2 0.1 4580:51 httpd
> 23698 apache 20 0 2293264 7524 4148 R 56.6 0.1 4582:17 httpd
> 23648 apache 20 0 2293264 7524 4148 R 55.9 0.1 4580:52 httpd
> 23668 apache 20 0 2293264 7524 4148 R 53.9 0.1 4581:23 httpd
> 1 root 20 0 128600 8592 5524 S 0.0 0.1 6:02.91 systemd
>

Can you get backtraces from a few such process with "pstack" or gdb?

https://httpd.apache.org/dev/debugging.html#backtrace

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: httpd 2.4 CPU spike after graceful restart [ In reply to ]
The gdb from some of the processes are:

--------
[root@prd-web2887652808b bin]# gdb httpd 23668
GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-30.amzn2.0.3
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from httpd...Reading symbols from
/usr/lib/debug/usr/sbin/httpd.debug...done.
done.
Attaching to program: /usr/sbin/httpd, process 23668
Reading symbols from target:/lib64/libpcre.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libpcre.so.1.2.0.debug...done.
done.
Reading symbols from target:/lib64/libselinux.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libselinux.so.1.debug...done.
done.
Reading symbols from target:/lib64/libaprutil-1.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libaprutil-1.so.0.6.1.debug...done.
done.
Reading symbols from target:/lib64/libcrypt.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libcrypt-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libexpat.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libexpat.so.1.6.0.debug...done.
done.
Reading symbols from target:/lib64/libapr-1.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libapr-1.so.0.6.3.debug...done.
done.
Reading symbols from target:/lib64/libpthread.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libpthread-2.26.so.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from target:/lib64/libdl.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libdl-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libc.so.6...Reading symbols from
/usr/lib/debug/usr/lib64/libc-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...Reading symbols
from /usr/lib/debug/usr/lib64/ld-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libuuid.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libuuid.so.1.3.0.debug...done.
done.
Reading symbols from target:/lib64/librt.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/librt-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libnss_sss.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libnss_sss.so.2.debug...done.
done.
Reading symbols from target:/lib64/libnss_files.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libnss_files-2.26.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_access_compat.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_access_compat.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_actions.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_actions.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_alias.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_alias.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_allowmethods.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_allowmethods.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_authn_core.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_authn_core.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_authz_core.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_authz_core.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_authz_host.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_authz_host.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_deflate.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_deflate.so.debug...done.
done.
Reading symbols from target:/lib64/libz.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_dir.so...Reading symbols
from /usr/lib/debug/usr/lib64/httpd/modules/mod_dir.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_dumpio.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_dumpio.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_env.so...Reading symbols
from /usr/lib/debug/usr/lib64/httpd/modules/mod_env.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_expires.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_expires.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_filter.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_filter.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_headers.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_headers.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_include.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_include.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_log_config.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_log_config.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_logio.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_logio.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_mime_magic.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_mime_magic.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_mime.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_mime.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_negotiation.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_negotiation.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_remoteip.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_remoteip.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_reqtimeout.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_reqtimeout.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_request.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_request.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_rewrite.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_rewrite.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_setenvif.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_setenvif.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_slotmem_shm.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_slotmem_shm.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_socache_shmcb.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_socache_shmcb.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_substitute.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_substitute.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_unixd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_unixd.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_vhost_alias.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_vhost_alias.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_watchdog.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_watchdog.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_mpm_worker.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_mpm_worker.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_proxy.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_proxy_connect.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_connect.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_proxy_express.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_express.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_proxy_http.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_systemd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_systemd.so.debug...done.
done.
Reading symbols from target:/lib64/libsystemd.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libsystemd.so.0.6.0.debug...done.
done.
Reading symbols from target:/lib64/libcap.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libcap.so.2.22.debug...done.
done.
Reading symbols from target:/lib64/libm.so.6...Reading symbols from
/usr/lib/debug/usr/lib64/libm-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/liblzma.so.5...Reading symbols from
/usr/lib/debug/usr/lib64/liblzma.so.5.2.2.debug...done.
done.
Reading symbols from target:/lib64/liblz4.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/liblz4.so.1.7.5.debug...done.
done.
Reading symbols from target:/lib64/libgcrypt.so.11...Reading symbols from
/usr/lib/debug/usr/lib64/libgcrypt.so.11.8.2.debug...done.
done.
Reading symbols from target:/lib64/libgpg-error.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libgpg-error.so.0.10.0.debug...done.
done.
Reading symbols from target:/lib64/libresolv.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libresolv-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libdw.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libdw-0.176.so.debug...done.
done.
Reading symbols from target:/lib64/libgcc_s.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libgcc_s-7-20180712.so.1.debug...done.
done.
Reading symbols from target:/lib64/libattr.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libattr.so.1.1.0.debug...done.
done.
Reading symbols from target:/lib64/libelf.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libelf-0.176.so.debug...done.
done.
Reading symbols from target:/lib64/libbz2.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libbz2.so.1.0.6.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_http2.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_http2.so.debug...done.
done.
Reading symbols from target:/lib64/libnghttp2.so.14...Reading symbols from
/usr/lib/debug/usr/lib64/libnghttp2.so.14.20.0.debug...done.
done.
Reading symbols from target:/lib64/libcrypto.so.10...Reading symbols from
/usr/lib/debug/usr/lib64/libcrypto.so.1.0.2k.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_proxy_http2.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http2.so.debug...done.
done.
get_next (slot=0x5591071acd18) at h2_workers.c:195
195 status = h2_fifo_try_peek(workers->mplxs, mplx_peek,
slot);
(gdb) where
#0 get_next (slot=0x5591071acd18) at h2_workers.c:195
#1 slot_run (thread=0x5591072441e0, wctx=0x5591071acd18) at
h2_workers.c:228
#2 0x00007fcf0690640b in start_thread (arg=0x7fced6bf5700) at
pthread_create.c:465
#3 0x00007fcf0643d09f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
--------
[root@prd-web2887652808b bin]# gdb httpd 23660
GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-30.amzn2.0.3
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from httpd...Reading symbols from
/usr/lib/debug/usr/sbin/httpd.debug...done.
done.
Attaching to program: /usr/sbin/httpd, process 23660
Reading symbols from target:/lib64/libpcre.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libpcre.so.1.2.0.debug...done.
done.
Reading symbols from target:/lib64/libselinux.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libselinux.so.1.debug...done.
done.
Reading symbols from target:/lib64/libaprutil-1.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libaprutil-1.so.0.6.1.debug...done.
done.
Reading symbols from target:/lib64/libcrypt.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libcrypt-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libexpat.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libexpat.so.1.6.0.debug...done.
done.
Reading symbols from target:/lib64/libapr-1.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libapr-1.so.0.6.3.debug...done.
done.
Reading symbols from target:/lib64/libpthread.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libpthread-2.26.so.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from target:/lib64/libdl.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libdl-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libc.so.6...Reading symbols from
/usr/lib/debug/usr/lib64/libc-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...Reading symbols
from /usr/lib/debug/usr/lib64/ld-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libuuid.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libuuid.so.1.3.0.debug...done.
done.
Reading symbols from target:/lib64/librt.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/librt-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libnss_sss.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libnss_sss.so.2.debug...done.
done.
Reading symbols from target:/lib64/libnss_files.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libnss_files-2.26.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_access_compat.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_access_compat.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_actions.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_actions.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_alias.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_alias.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_allowmethods.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_allowmethods.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_authn_core.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_authn_core.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_authz_core.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_authz_core.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_authz_host.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_authz_host.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_deflate.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_deflate.so.debug...done.
done.
Reading symbols from target:/lib64/libz.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_dir.so...Reading symbols
from /usr/lib/debug/usr/lib64/httpd/modules/mod_dir.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_dumpio.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_dumpio.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_env.so...Reading symbols
from /usr/lib/debug/usr/lib64/httpd/modules/mod_env.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_expires.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_expires.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_filter.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_filter.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_headers.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_headers.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_include.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_include.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_log_config.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_log_config.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_logio.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_logio.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_mime_magic.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_mime_magic.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_mime.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_mime.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_negotiation.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_negotiation.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_remoteip.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_remoteip.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_reqtimeout.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_reqtimeout.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_request.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_request.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_rewrite.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_rewrite.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_setenvif.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_setenvif.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_slotmem_shm.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_slotmem_shm.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_socache_shmcb.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_socache_shmcb.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_substitute.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_substitute.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_unixd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_unixd.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_vhost_alias.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_vhost_alias.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_watchdog.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_watchdog.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_mpm_worker.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_mpm_worker.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_proxy.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_proxy_connect.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_connect.so.debug...done.
done.
Reading symbols from
target:/etc/httpd/modules/mod_proxy_express.so...Reading symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_express.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_proxy_http.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http.so.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_systemd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_systemd.so.debug...done.
done.
Reading symbols from target:/lib64/libsystemd.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libsystemd.so.0.6.0.debug...done.
done.
Reading symbols from target:/lib64/libcap.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libcap.so.2.22.debug...done.
done.
Reading symbols from target:/lib64/libm.so.6...Reading symbols from
/usr/lib/debug/usr/lib64/libm-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/liblzma.so.5...Reading symbols from
/usr/lib/debug/usr/lib64/liblzma.so.5.2.2.debug...done.
done.
Reading symbols from target:/lib64/liblz4.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/liblz4.so.1.7.5.debug...done.
done.
Reading symbols from target:/lib64/libgcrypt.so.11...Reading symbols from
/usr/lib/debug/usr/lib64/libgcrypt.so.11.8.2.debug...done.
done.
Reading symbols from target:/lib64/libgpg-error.so.0...Reading symbols from
/usr/lib/debug/usr/lib64/libgpg-error.so.0.10.0.debug...done.
done.
Reading symbols from target:/lib64/libresolv.so.2...Reading symbols from
/usr/lib/debug/usr/lib64/libresolv-2.26.so.debug...done.
done.
Reading symbols from target:/lib64/libdw.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libdw-0.176.so.debug...done.
done.
Reading symbols from target:/lib64/libgcc_s.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libgcc_s-7-20180712.so.1.debug...done.
done.
Reading symbols from target:/lib64/libattr.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libattr.so.1.1.0.debug...done.
done.
Reading symbols from target:/lib64/libelf.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libelf-0.176.so.debug...done.
done.
Reading symbols from target:/lib64/libbz2.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/libbz2.so.1.0.6.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_http2.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_http2.so.debug...done.
done.
Reading symbols from target:/lib64/libnghttp2.so.14...Reading symbols from
/usr/lib/debug/usr/lib64/libnghttp2.so.14.20.0.debug...done.
done.
Reading symbols from target:/lib64/libcrypto.so.10...Reading symbols from
/usr/lib/debug/usr/lib64/libcrypto.so.1.0.2k.debug...done.
done.
Reading symbols from target:/etc/httpd/modules/mod_proxy_http2.so...Reading
symbols from
/usr/lib/debug/usr/lib64/httpd/modules/mod_proxy_http2.so.debug...done.
done.
get_next (slot=0x5591071acd98) at h2_workers.c:196
196 if (status == APR_EOF) {
(gdb) where
#0 get_next (slot=0x5591071acd98) at h2_workers.c:196
#1 slot_run (thread=0x559107244030, wctx=0x5591071acd98) at
h2_workers.c:228
#2 0x00007fcf0690640b in start_thread (arg=0x7fced7bf7700) at
pthread_create.c:465
#3 0x00007fcf0643d09f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
---------
fifo_peek (block=0, ctx=ctx@entry=0x5591071acf58, fn=fn@entry=0x7fceff358fd0
<mplx_peek>, fifo=0x5591071ac7a0) at h2_util.c:767
767 return APR_EOF;
(gdb) where
#0 fifo_peek (block=0, ctx=ctx@entry=0x5591071acf58,
fn=fn@entry=0x7fceff358fd0
<mplx_peek>, fifo=0x5591071ac7a0) at h2_util.c:767
#1 h2_fifo_try_peek (fifo=0x5591071ac7a0, fn=fn@entry=0x7fceff358fd0
<mplx_peek>, ctx=ctx@entry=0x5591071acf58) at h2_util.c:792
#2 0x00007fceff358e3d in get_next (slot=0x5591071acf58) at h2_workers.c:195
#3 slot_run (thread=0x559107243a48, wctx=0x5591071acf58) at
h2_workers.c:228
#4 0x00007fcf0690640b in start_thread (arg=0x7fcedb3fe700) at
pthread_create.c:465
#5 0x00007fcf0643d09f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
-----------
slot_run (thread=0x559107243da8, wctx=0x5591071ace58) at h2_workers.c:225
225 while (!slot->aborted) {
(gdb) where
#0 slot_run (thread=0x559107243da8, wctx=0x5591071ace58) at
h2_workers.c:225
#1 0x00007fcf0690640b in start_thread (arg=0x7fced93fa700) at
pthread_create.c:465
#2 0x00007fcf0643d09f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
------------
I also ran gdb on the parent process:
Attaching to program: /usr/sbin/httpd, process 22231
[New LWP 23620]
[New LWP 23636]
[New LWP 23648]
[New LWP 23660]
[New LWP 23664]
[New LWP 23668]
[New LWP 23677]
[New LWP 23698]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fcf0690bd63 in futex_wait (private=<optimized out>, expected=12,
futex_word=0x5591072435e4) at ../sysdeps/unix/sysv/linux/futex-internal.h:61
61 int err = lll_futex_timed_wait (futex_word, expected, NULL,
private);
(gdb) where
#0 0x00007fcf0690bd63 in futex_wait (private=<optimized out>, expected=12,
futex_word=0x5591072435e4) at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1 futex_wait_simple (private=<optimized out>, expected=12,
futex_word=0x5591072435e4) at ../sysdeps/nptl/futex-internal.h:135
#2 __pthread_cond_destroy (cond=0x5591072435c0) at
pthread_cond_destroy.c:54
#3 0x00007fcf06b3bcee in run_cleanups (cref=<optimized out>) at
memory/unix/apr_pools.c:2629
#4 apr_pool_destroy (pool=0x5591071ac6b8) at memory/unix/apr_pools.c:1000
#5 0x00007fcf06b3bccd in apr_pool_destroy (pool=0x55910711a918) at
memory/unix/apr_pools.c:997
#6 0x00007fcf01e86e55 in clean_child_exit (code=0) at worker.c:436
#7 0x00007fcf01e8795e in child_main (child_num_arg=child_num_arg@entry=5,
child_bucket=child_bucket@entry=0) at worker.c:1197
#8 0x00007fcf01e87ffb in make_child (s=0x559107069650, slot=slot@entry=5,
bucket=0) at worker.c:1328
#9 0x00007fcf01e8806d in startup_children (number_to_start=16,
number_to_start@entry=20) at worker.c:1355
#10 0x00007fcf01e88c03 in server_main_loop (num_buckets=1,
remaining_children_to_start=20) at worker.c:1669
#11 worker_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized
#12 0x0000559105d346da in ap_run_mpm (pconf=0x5591070403a8,
plog=0x55910706d5c8, s=0x559107069650) at mpm_common.c:94
#13 0x0000559105d2d01c in main (argc=<optimized out>, argv=<optimized out>)
at main.c:819


On Thu, Jun 3, 2021 at 6:30 PM Eric Covener <covener@gmail.com> wrote:

> On Thu, Jun 3, 2021 at 6:23 AM Jay Ce <jceh12021@gmail.com> wrote:
> >
> > Hi,
> >
> > I noticed that sometimes when performing a graceful restart of httpd,
> the CPU of the httpd process spikes and does not come down. This happens
> more frequently if the web server is under load but can happen even when
> not serving traffic. Has anyone else encountered this problem? And what can
> I do to resolve this?
> >
> > I did not notice any error messages in the error log and only see the
> following:
> > Sat May 29 03:33:01.424748 2021] [mpm_worker:notice] [pid 2773:tid
> 140527163445824] AH00297: SIGUSR1 received. Doing graceful restart
> > AH00558: httpd: Could not reliably determine the server's fully
> qualified domain name, using 127.0.0.1. Set the 'ServerName' directive
> globally to suppress this message
> > [Sat May 29 03:33:01.485368 2021] [http2:warn] [pid 2773:tid
> 140527163445824] AH02951: mod_ssl does not seem to be enabled
> > [Sat May 29 03:33:01.485953 2021] [mpm_worker:notice] [pid 2773:tid
> 140527163445824] AH00292: Apache/2.4.46 () configured -- resuming normal
> operations
> > [Sat May 29 03:33:01.485964 2021] [core:notice] [pid 2773:tid
> 140527163445824] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
> > [Sat May 29 03:33:01.486045 2021] [mpm_worker:warn] [pid 2773:tid
> 140527163445824] AH00291: long lost child came home! (pid 25489)
> >
> >
> > From performing a top
> > top - 17:44:22 up 5 days, 16:04, 1 user, load average: 7.00, 7.13, 8.27
> > Threads: 384 total, 8 running, 316 sleeping, 0 stopped, 1 zombie
> > %Cpu(s): 99.8 us, 0.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si,
> 0.0 st
> > KiB Mem : 7807268 total, 4124256 free, 543316 used, 3139696
> buff/cache
> > KiB Swap: 1310716 total, 1310716 free, 0 used. 6949164 avail
> Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> > 23677 apache 20 0 2293264 7524 4148 R 58.6 0.1 4582:47 httpd
> > 23664 apache 20 0 2293264 7524 4148 R 58.2 0.1 4581:57 httpd
> > 23636 apache 20 0 2293264 7524 4148 R 57.6 0.1 4582:16 httpd
> > 23660 apache 20 0 2293264 7524 4148 R 57.2 0.1 4580:51 httpd
> > 23698 apache 20 0 2293264 7524 4148 R 56.6 0.1 4582:17 httpd
> > 23648 apache 20 0 2293264 7524 4148 R 55.9 0.1 4580:52 httpd
> > 23668 apache 20 0 2293264 7524 4148 R 53.9 0.1 4581:23 httpd
> > 1 root 20 0 128600 8592 5524 S 0.0 0.1 6:02.91
> systemd
> >
>
> Can you get backtraces from a few such process with "pstack" or gdb?
>
> https://httpd.apache.org/dev/debugging.html#backtrace
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>