Mailing List Archive

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #2 from chris <exxos21@googlemail.com> ---
Added a new image. It show a connection (mostly they are from AWS) where it is
stuck on "W" on the same link. No figures change other than the CPU time which
just goes up constantly, but also consumes 100% CPU while in that state.

Also I have seen the same stuck on "G" where the CPU time constantly goes up,
but the connection never seems to complete. Though I don't understand why it
would consume 100% CPU in the first place.

2 URLs last night were from AWS. I had a huge DoS attack from AWS a few weeks
ago. I blocked all the offending IPs. Now just 2 of these connections are
causing the server CPU to max out.

The offending URLs are file.php from PHPBB forum. I killed the connections last
night, then a moment later they came back with 100% CPU again. So I renamed
file.php and no CPU or connection issues since. Everything seems to be running
fine currently and as expected.

The files which are downloading are nothing but images. normally 2MB or below
in size. I checked the URLs manually and they load quickly and terminate the
connection as expected.

There is nothing in the logs or status pages to indicate any DoS attacks. The
server is only server about 1 connection a second. The server can cope with a
much higher load easily anyway.

So the problem only seems to happen with AWS IPs from what I can tell
currently. I have no idea why. Or why "hung connections" take 100% CPU.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #3 from chris <exxos21@googlemail.com> ---
All I can really tell is, A connection opens to download something, mostly
small images. The connection sends about 50-60KB then "stalls". It then sits in
that state until its manually killed.

PHP timeout is 900 seconds. Apache timeout is 300 seconds. I also did the mod
where if no activity for 20 seconds it should drop the connection.

But I have seen the problem connections open for literally hours. There is no
data transfer at all. If left in that state it seems to be when Apache reports
(using TOP) 100% CPU.

So again, I don't understand what is going on , why the timeouts seem not to be
working, and why "idle connections" end up eating up 100% CPU eventually.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #4 from chris <exxos21@googlemail.com> ---
All I can really tell is, A connection opens to download something, mostly
small images. The connection sends about 50-60KB then "stalls". It then sits in
that state until its manually killed.

PHP timeout is 900 seconds. Apache timeout is 300 seconds. I also did the mod
where if no activity for 20 seconds it should drop the connection.

But I have seen the problem connections open for literally hours. There is no
data transfer at all. If left in that state it seems to be when Apache reports
(using TOP) 100% CPU.

So again, I don't understand what is going on , why the timeouts seem not to be
working, and why "idle connections" end up eating up 100% CPU eventually.

I will turn off "keepalive" tomorrow and see if that helps.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

Eric Covener <covener@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO

--- Comment #5 from Eric Covener <covener@gmail.com> ---
can you post a backtrace of a spinning process?

https://httpd.apache.org/dev/debugging.html

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #6 from chris <exxos21@googlemail.com> ---
# gdb -p 182047
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 182047
[New LWP 182048]
[New LWP 182049]
[New LWP 182050]
[New LWP 182051]
[New LWP 182052]
[New LWP 182053]
[New LWP 182054]
[New LWP 182055]
[New LWP 182056]
[New LWP 182057]
[New LWP 182059]
[New LWP 182060]
[New LWP 182061]
[New LWP 182062]
[New LWP 182063]
[New LWP 182064]
[New LWP 182065]
[New LWP 182066]
[New LWP 182067]
[New LWP 182068]
[New LWP 182069]
[New LWP 183166]
[New LWP 183167]
[New LWP 183168]
[New LWP 187880]
[New LWP 187881]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--c
__libc_read (nbytes=1, buf=0x7ffd2e6e8e63, fd=7) at
../sysdeps/unix/sysv/linux/read.c:26
26 ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) bt
#0 __libc_read (nbytes=1, buf=0x7ffd2e6e8e63, fd=7) at
../sysdeps/unix/sysv/linux/read.c:26
#1 __libc_read (fd=7, buf=0x7ffd2e6e8e63, nbytes=1) at
../sysdeps/unix/sysv/linux/read.c:24
#2 0x0000559bc199f58b in ap_mpm_podx_check ()
#3 0x00007f5ccdc22387 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#4 0x00007f5ccdc227ba in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdc22835 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x00007f5ccdc23379 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#7 0x0000559bc1971de8 in ap_run_mpm ()
#8 0x0000559bc19697f9 in main ()
(gdb) where
#0 __libc_read (nbytes=1, buf=0x7ffd2e6e8e63, fd=7) at
../sysdeps/unix/sysv/linux/read.c:26
#1 __libc_read (fd=7, buf=0x7ffd2e6e8e63, nbytes=1) at
../sysdeps/unix/sysv/linux/read.c:24
#2 0x0000559bc199f58b in ap_mpm_podx_check ()
#3 0x00007f5ccdc22387 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#4 0x00007f5ccdc227ba in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdc22835 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x00007f5ccdc23379 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#7 0x0000559bc1971de8 in ap_run_mpm ()
#8 0x0000559bc19697f9 in main ()
(gdb) thread apply all bt

Thread 27 (Thread 0x7f5c9f7fe700 (LWP 187881)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5c9f7fde10, clockid=<optimized out>, expected=0,
futex_word=0x7f5ccdc0ace8) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5c9f7fde10, clockid=<optimized
out>, mutex=0x7f5ccdc0a1c8, cond=0x7f5ccdc0acc0) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5ccdc0acc0, mutex=0x7f5ccdc0a1c8,
abstime=0x7f5c9f7fde10) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd913681 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 26 (Thread 0x7f5ca4ff9700 (LWP 187880)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5ca4ff8e10, clockid=<optimized out>, expected=0,
futex_word=0x7f5ccdc0ac94) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5ca4ff8e10, clockid=<optimized
out>, mutex=0x7f5ccdc0a1c8, cond=0x7f5ccdc0ac68) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5ccdc0ac68, mutex=0x7f5ccdc0a1c8,
abstime=0x7f5ca4ff8e10) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd913681 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 25 (Thread 0x7f5ca57fa700 (LWP 183168)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5ca57f9e10, clockid=<optimized out>, expected=0,
futex_word=0x7f5ccdc0ac38) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5ca57f9e10, clockid=<optimized
out>, mutex=0x7f5ccdc0a1c8, cond=0x7f5ccdc0ac10) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5ccdc0ac10, mutex=0x7f5ccdc0a1c8,
abstime=0x7f5ca57f9e10) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd913681 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 24 (Thread 0x7f5ca5ffb700 (LWP 183167)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5ca5ffae10, clockid=<optimized out>, expected=0,
futex_word=0x7f5ccdc0abe0) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5ca5ffae10, clockid=<optimized
out>, mutex=0x7f5ccdc0a1c8, cond=0x7f5ccdc0abb8) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5ccdc0abb8, mutex=0x7f5ccdc0a1c8,
abstime=0x7f5ca5ffae10) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd913681 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
--Type <RET> for more, q to quit, c to continue without paging--

Thread 23 (Thread 0x7f5ca67fc700 (LWP 183166)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5ca67f66b0, clockid=<optimized out>, expected=0,
futex_word=0x7f5cc451a1f8) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5ca67f66b0, clockid=<optimized
out>, mutex=0x7f5cc451a180, cond=0x7f5cc451a1d0) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5cc451a1d0, mutex=0x7f5cc451a180,
abstime=0x7f5ca67f66b0) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd8f1d50 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccd8f7abe in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#6 0x00007f5ccd8f8c61 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#7 0x0000559bc1978495 in ap_content_length_filter ()
#8 0x00007f5ccd8b6f1e in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#9 0x00007f5ccd8b7f2c in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#10 0x00007f5ccd86a31b in proxy_run_scheme_handler () from
target:/usr/lib/apache2/modules/mod_proxy.so
#11 0x00007f5ccd86c40b in ?? () from
target:/usr/lib/apache2/modules/mod_proxy.so
#12 0x0000559bc1991ec8 in ap_run_handler ()
#13 0x0000559bc1992476 in ap_invoke_handler ()
#14 0x0000559bc19aace3 in ap_process_async_request ()
#15 0x0000559bc19aaeb2 in ap_process_request ()
#16 0x00007f5ccd8f7239 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#17 0x0000559bc199bc68 in ap_run_process_connection ()
#18 0x00007f5ccd8f7daf in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#19 0x00007f5ccd9137e7 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#20 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#21 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 22 (Thread 0x7f5ca6ffd700 (LWP 182069)):
#0 0x00007f5ccdebe46e in epoll_wait (epfd=18, events=0x7f5ccd708488,
maxevents=23, timeout=9996) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007f5ccdfdc98e in ?? () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f5ccdc247f8 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#3 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#4 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7f5ca77fe700 (LWP 182068)):
#0 0x00007f5ccdeb199f in __GI___poll (fds=0x7f5ca77fd990, nfds=1,
timeout=300000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f5ccdfdd6fe in apr_poll () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x0000559bc198d265 in ap_core_output_filter ()
#3 0x00007f5ccd6b51fa in ?? () from target:/usr/lib/apache2/modules/mod_ssl.so
#4 0x00007f5ccd6b1fd2 in ?? () from target:/usr/lib/apache2/modules/mod_ssl.so
#5 0x00007f5ccd8f4cd5 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
--Type <RET> for more, q to quit, c to continue without paging--
#6 0x00007f5ccd9088b4 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#7 0x00007f5ccd8f3551 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#8 0x00007f5ccd8f39ab in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#9 0x0000559bc199bc68 in ap_run_process_connection ()
#10 0x00007f5ccdc216e5 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#11 0x00007f5ccdc259fd in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#12 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#13 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7f5ca7fff700 (LWP 182067)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccd708160) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccd7080e8,
cond=0x7f5ccd708138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccd708138, mutex=0x7f5ccd7080e8) at
pthread_cond_wait.c:647
#3 0x0000559bc19a046d in ap_queue_pop_something ()
#4 0x00007f5ccdc256fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7f5c9ffff700 (LWP 182066)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccd708164) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccd7080e8,
cond=0x7f5ccd708138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccd708138, mutex=0x7f5ccd7080e8) at
pthread_cond_wait.c:647
#3 0x0000559bc19a046d in ap_queue_pop_something ()
#4 0x00007f5ccdc256fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7f5cc4d79700 (LWP 182065)):
#0 0x00007f5ccd8cab88 in ?? () from
target:/lib/x86_64-linux-gnu/libnghttp2.so.14
#1 0x00007f5ccd8cbca9 in nghttp2_session_send () from
target:/lib/x86_64-linux-gnu/libnghttp2.so.14
#2 0x00007f5ccd908350 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#3 0x00007f5ccd90894d in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#4 0x00007f5ccd8f3551 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccd8f39ab in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#6 0x0000559bc199bc68 in ap_run_process_connection ()
#7 0x00007f5ccdc216e5 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#8 0x00007f5ccdc259fd in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#9 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#10 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

--Type <RET> for more, q to quit, c to continue without paging--
Thread 17 (Thread 0x7f5cc557a700 (LWP 182064)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccd708160) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccd7080e8,
cond=0x7f5ccd708138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccd708138, mutex=0x7f5ccd7080e8) at
pthread_cond_wait.c:647
#3 0x0000559bc19a046d in ap_queue_pop_something ()
#4 0x00007f5ccdc256fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f5cc5d7b700 (LWP 182063)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccd708160) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccd7080e8,
cond=0x7f5ccd708138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccd708138, mutex=0x7f5ccd7080e8) at
pthread_cond_wait.c:647
#3 0x0000559bc19a046d in ap_queue_pop_something ()
#4 0x00007f5ccdc256fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f5cc657c700 (LWP 182062)):
#0 0x00007f5ccdeb199f in __GI___poll (fds=0x7f5cc657a550, nfds=1,
timeout=300000) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f5ccdfdd6fe in apr_poll () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x0000559bc198d265 in ap_core_output_filter ()
#3 0x00007f5ccd6b1888 in ?? () from target:/usr/lib/apache2/modules/mod_ssl.so
#4 0x00007f5ccd6b1bcd in ?? () from target:/usr/lib/apache2/modules/mod_ssl.so
#5 0x00007f5ccd9d066e in ?? () from
target:/lib/x86_64-linux-gnu/libcrypto.so.1.1
#6 0x00007f5ccd9cf684 in ?? () from
target:/lib/x86_64-linux-gnu/libcrypto.so.1.1
#7 0x00007f5ccd9cfb47 in BIO_write () from
target:/lib/x86_64-linux-gnu/libcrypto.so.1.1
#8 0x00007f5ccd7c8dde in ?? () from target:/lib/x86_64-linux-gnu/libssl.so.1.1
#9 0x00007f5ccd7c9cd9 in ?? () from target:/lib/x86_64-linux-gnu/libssl.so.1.1
#10 0x00007f5ccd7c9f75 in ?? () from target:/lib/x86_64-linux-gnu/libssl.so.1.1
#11 0x00007f5ccd7dccd9 in ?? () from target:/lib/x86_64-linux-gnu/libssl.so.1.1
#12 0x00007f5ccd7dce17 in SSL_write () from
target:/lib/x86_64-linux-gnu/libssl.so.1.1
#13 0x00007f5ccd6b5365 in ?? () from target:/usr/lib/apache2/modules/mod_ssl.so
#14 0x00007f5ccd6b1fd2 in ?? () from target:/usr/lib/apache2/modules/mod_ssl.so
#15 0x00007f5ccd8f4cd5 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#16 0x00007f5ccd908429 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#17 0x00007f5ccd90885f in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#18 0x00007f5ccd8f3551 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#19 0x00007f5ccd8f39ab in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#20 0x0000559bc199bc68 in ap_run_process_connection ()
--Type <RET> for more, q to quit, c to continue without paging--
#21 0x00007f5ccdc216e5 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#22 0x00007f5ccdc259fd in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#23 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#24 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7f5cc6d7d700 (LWP 182061)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccd708160) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccd7080e8,
cond=0x7f5ccd708138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccd708138, mutex=0x7f5ccd7080e8) at
pthread_cond_wait.c:647
#3 0x0000559bc19a046d in ap_queue_pop_something ()
#4 0x00007f5ccdc256fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f5cc757e700 (LWP 182060)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccd708160) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccd7080e8,
cond=0x7f5ccd708138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccd708138, mutex=0x7f5ccd7080e8) at
pthread_cond_wait.c:647
#3 0x0000559bc19a046d in ap_queue_pop_something ()
#4 0x00007f5ccdc256fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f5cc7d7f700 (LWP 182059)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccd708164) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccd7080e8,
cond=0x7f5ccd708138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccd708138, mutex=0x7f5ccd7080e8) at
pthread_cond_wait.c:647
#3 0x0000559bc19a046d in ap_queue_pop_something ()
#4 0x00007f5ccdc256fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f5cc8d81700 (LWP 182057)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5cc8d7b6b0, clockid=<optimized out>, expected=0,
futex_word=0x7f5cc43b51f8) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5cc8d7b6b0, clockid=<optimized
out>, mutex=0x7f5cc43b5180, cond=0x7f5cc43b51d0) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5cc43b51d0, mutex=0x7f5cc43b5180,
abstime=0x7f5cc8d7b6b0) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd8f1d50 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccd8f7abe in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
--Type <RET> for more, q to quit, c to continue without paging--
#6 0x00007f5ccd8f8c61 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#7 0x0000559bc1978495 in ap_content_length_filter ()
#8 0x00007f5ccd8b6f1e in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#9 0x00007f5ccd8b7f2c in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#10 0x00007f5ccd86a31b in proxy_run_scheme_handler () from
target:/usr/lib/apache2/modules/mod_proxy.so
#11 0x00007f5ccd86c40b in ?? () from
target:/usr/lib/apache2/modules/mod_proxy.so
#12 0x0000559bc1991ec8 in ap_run_handler ()
#13 0x0000559bc1992476 in ap_invoke_handler ()
#14 0x0000559bc19aace3 in ap_process_async_request ()
#15 0x0000559bc19aaeb2 in ap_process_request ()
#16 0x00007f5ccd8f7239 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#17 0x0000559bc199bc68 in ap_run_process_connection ()
#18 0x00007f5ccd8f7daf in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#19 0x00007f5ccd9137e7 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#20 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#21 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f5cc9582700 (LWP 182056)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5cc957c6b0, clockid=<optimized out>, expected=0,
futex_word=0x7f5cc445e1fc) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5cc957c6b0, clockid=<optimized
out>, mutex=0x7f5cc445e180, cond=0x7f5cc445e1d0) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5cc445e1d0, mutex=0x7f5cc445e180,
abstime=0x7f5cc957c6b0) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd8f1d50 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccd8f7abe in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#6 0x00007f5ccd8f8c61 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#7 0x0000559bc1978495 in ap_content_length_filter ()
#8 0x00007f5ccd8b6f1e in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#9 0x00007f5ccd8b7f2c in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#10 0x00007f5ccd86a31b in proxy_run_scheme_handler () from
target:/usr/lib/apache2/modules/mod_proxy.so
#11 0x00007f5ccd86c40b in ?? () from
target:/usr/lib/apache2/modules/mod_proxy.so
#12 0x0000559bc1991ec8 in ap_run_handler ()
#13 0x0000559bc1992476 in ap_invoke_handler ()
#14 0x0000559bc19aace3 in ap_process_async_request ()
#15 0x0000559bc19aaeb2 in ap_process_request ()
#16 0x00007f5ccd8f7239 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#17 0x0000559bc199bc68 in ap_run_process_connection ()
#18 0x00007f5ccd8f7daf in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#19 0x00007f5ccd9137e7 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#20 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
--Type <RET> for more, q to quit, c to continue without paging--
#21 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f5cc9d83700 (LWP 182055)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccdc0aa80) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccdc0a1c8,
cond=0x7f5ccdc0aa58) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccdc0aa58, mutex=0x7f5ccdc0a1c8) at
pthread_cond_wait.c:647
#3 0x00007f5ccd913651 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#4 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f5cca584700 (LWP 182054)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccdc0aa28) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccdc0a1c8,
cond=0x7f5ccdc0aa00) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccdc0aa00, mutex=0x7f5ccdc0a1c8) at
pthread_cond_wait.c:647
#3 0x00007f5ccd913651 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#4 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f5ccad85700 (LWP 182053)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>,
abstime=0x7f5ccad7f6b0, clockid=<optimized out>, expected=0,
futex_word=0x7f5cc43d61f8) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7f5ccad7f6b0, clockid=<optimized
out>, mutex=0x7f5cc43d6180, cond=0x7f5cc43d61d0) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x7f5cc43d61d0, mutex=0x7f5cc43d6180,
abstime=0x7f5ccad7f6b0) at pthread_cond_wait.c:665
#3 0x00007f5ccdfd4a3b in apr_thread_cond_timedwait () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#4 0x00007f5ccd8f1d50 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#5 0x00007f5ccd8f7abe in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#6 0x00007f5ccd8f8c61 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#7 0x0000559bc1978495 in ap_content_length_filter ()
#8 0x00007f5ccd8b6f1e in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#9 0x00007f5ccd8b7f2c in ?? () from
target:/usr/lib/apache2/modules/mod_proxy_fcgi.so
#10 0x00007f5ccd86a31b in proxy_run_scheme_handler () from
target:/usr/lib/apache2/modules/mod_proxy.so
#11 0x00007f5ccd86c40b in ?? () from
target:/usr/lib/apache2/modules/mod_proxy.so
#12 0x0000559bc1991ec8 in ap_run_handler ()
#13 0x0000559bc1992476 in ap_invoke_handler ()
#14 0x0000559bc19aace3 in ap_process_async_request ()
#15 0x0000559bc19aaeb2 in ap_process_request ()
#16 0x00007f5ccd8f7239 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#17 0x0000559bc199bc68 in ap_run_process_connection ()
#18 0x00007f5ccd8f7daf in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#19 0x00007f5ccd9137e7 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
--Type <RET> for more, q to quit, c to continue without paging--
#20 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#21 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f5ccb586700 (LWP 182052)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccdc0a97c) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccdc0a1c8,
cond=0x7f5ccdc0a950) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccdc0a950, mutex=0x7f5ccdc0a1c8) at
pthread_cond_wait.c:647
#3 0x00007f5ccd913651 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#4 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f5ccbd87700 (LWP 182051)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccdc0a924) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccdc0a1c8,
cond=0x7f5ccdc0a8f8) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccdc0a8f8, mutex=0x7f5ccdc0a1c8) at
pthread_cond_wait.c:647
#3 0x00007f5ccd913651 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#4 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f5ccc588700 (LWP 182050)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccdc0a8cc) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccdc0a1c8,
cond=0x7f5ccdc0a8a0) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccdc0a8a0, mutex=0x7f5ccdc0a1c8) at
pthread_cond_wait.c:647
#3 0x00007f5ccd913651 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#4 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f5cccd89700 (LWP 182049)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccdc0a870) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccdc0a1c8,
cond=0x7f5ccdc0a848) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccdc0a848, mutex=0x7f5ccdc0a1c8) at
pthread_cond_wait.c:647
#3 0x00007f5ccd913651 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
#4 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f5ccd58a700 (LWP 182048)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7f5ccdc0a81c) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f5ccdc0a1c8,
cond=0x7f5ccdc0a7f0) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7f5ccdc0a7f0, mutex=0x7f5ccdc0a1c8) at
pthread_cond_wait.c:647
#3 0x00007f5ccd913651 in ?? () from
target:/usr/lib/apache2/modules/mod_http2.so
--Type <RET> for more, q to quit, c to continue without paging--
#4 0x00007f5ccdf99609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007f5ccdebe133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f5ccdd1a500 (LWP 182047)):
#0 __libc_read (nbytes=1, buf=0x7ffd2e6e8e63, fd=7) at
../sysdeps/unix/sysv/linux/read.c:26
#1 __libc_read (fd=7, buf=0x7ffd2e6e8e63, nbytes=1) at
../sysdeps/unix/sysv/linux/read.c:24
#2 0x0000559bc199f58b in ap_mpm_podx_check ()
#3 0x00007f5ccdc22387 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#4 0x00007f5ccdc227ba in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f5ccdc22835 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x00007f5ccdc23379 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#7 0x0000559bc1971de8 in ap_run_mpm ()
#8 0x0000559bc19697f9 in main ()
(gdb)



I have switched to prefork just to see if that changes anything. Now I see
"Keep alive" connections poppingup, and "close connections" in the
server-status page. I didn't using event. So something has changed. I need to
leave it on prefork for a few hours to see if the problem returns.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #7 from Stefan Eissing <stefan@eissing.org> ---
Thanks for the backtrace. In it, I see one http2 connection that tries to send
data to the client. All other thread are in `cond_wait()`, afaict.

Assuming this thread is causing the 100%, I looked at the code path involved. I
have found a missing return code check that *may* be involved here, causing the
http2 session to try on sending on an aborted connection.

Would you be able to build a new mod_http2 from source? I'll attach a patch
here, but you can also build from <https://github.com/icing/mod_h2> using the
master branch where the patch is applied.

As a note: switching to prefork will auto-disable the http2 protocol. If this
runs without problems, you could also go back to your original configuration
and disable HTTP/2.

Regards,
Stefan

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #8 from Stefan Eissing <stefan@eissing.org> ---
Created attachment 38572
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38572&action=edit
proposed patch to mod_http2

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #9 from chris <exxos21@googlemail.com> ---
Thanks, I don't know how to compile stuff, but I might know someone who can do
it for me.

Indeed the connections which get stuck never seem to close, I have had them
open for hours. Its those which get stuck consume max CPU usage.

Prefork seems to be working fine for several hours. I see "K" and "C" now in
server status whereas before I never did.

Hopefully I can try your patch to see if it resolves the problem, Thanks.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

chris <exxos21@googlemail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW

--- Comment #10 from chris <exxos21@googlemail.com> ---
I am now running the patched http2 module.

Will have to stress test over the next several hours.

Though currently there still seems to be connections stuck on "W" for over 500
seconds. They do seem to close eventually so I'm not sure if they're hitting
the php timeout.

Though with prefork, I have the time out as 10 seconds, this seems to work fine
with pre-four, but all the time out seem to be completely ignored with event ?

Also I've had a couple reports already that the server starts sporadically
running slowly. Currently there is no high CPU usage. And basically 99% free
/open connections. So I don't know where the slowdown is coming from as
everything seemed to run perfectly fine with prefork.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #11 from chris <exxos21@googlemail.com> ---
Created attachment 38574
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38574&action=edit
Patched hung W connections

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

chris <exxos21@googlemail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO

--- Comment #12 from chris <exxos21@googlemail.com> ---
Things seemed to have settled down somewhat. I have been running
ApacheBenchmark with up to 50 concurrent connections and it seems to be
processing and closing them fine.

I think the slowdowns are originating from the SQL server not Apache.

I need to leave it running for a few days to make sure it is stable. How things
were before it would randomly start maxing out the CPU after a few hours or a
few days.

So I will report back in a few days time if things are still working correctly
or not.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #13 from Stefan Eissing <stefan@eissing.org> ---
Thanks for testing this. Looking forward to not hearing from you for a long
time!

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #14 from Stefan Eissing <stefan@eissing.org> ---
My humour sometimes gets me carried away. I hope we have found the problem that
is causing issues for you is what I meant to say.????

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #15 from Pascal <pascal.christen@hostpoint.ch> ---
We also have problems like this (since Apache 2.4.55 - FreeBSD) where
afterwards processes are in the "uwait" state and are using 100% CPU. I'm
currently testing the patch you posted and will give you feedback by the end of
the week.

Greetings Pascal

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

Pascal <pascal.christen@hostpoint.ch> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |pascal.christen@hostpoint.c
| |h

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #16 from chris <exxos21@googlemail.com> ---
(In reply to Stefan Eissing from comment #14)
> My humour sometimes gets me carried away. I hope we have found the problem
> that is causing issues for you is what I meant to say.????

Yes understood. Thought it was funny, I generally have that effect on people as
standard anyway lol

Still no 100% CPU usage. I was bashing it with connections for a long time
yesterday and it recovered fine. I need to leave it running for a few days to
be 100% sure its solved. So far so good though.

Though I am still not sure the timeouts are working as connections still get
stuck over over 400 seconds since anything happened. They do close eventually.
So I'm not sure if that's a issue or not ?

On prefork the timeouts are 10 seconds and I never saw a connection open for
more than 10 seconds.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #17 from chris <exxos21@googlemail.com> ---
Created attachment 38575
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38575&action=edit
G-connections hung

Another example how connections seem to get"hung". Currently on "G". Though
also happens a lot with "W".

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #18 from Eric Covener <covener@gmail.com> ---
(In reply to chris from comment #17)
> Created attachment 38575 [details]
> G-connections hung
>
> Another example how connections seem to get"hung". Currently on "G". Though
> also happens a lot with "W".

The G and W are probably the same situation, with the difference that the
process involved in G are trying to exit (and held up by a hung request).

If you see 'G', find the process ID and get backtraces a few seconds apart.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #19 from chris <exxos21@googlemail.com> ---
(In reply to Eric Covener from comment #18)
> The G and W are probably the same situation, with the difference that the
> process involved in G are trying to exit (and held up by a hung request).
>
> If you see 'G', find the process ID and get backtraces a few seconds apart.

A couple were hung on G, and this is all which came up.


Attaching to process 175367
[New LWP 175368]
[New LWP 175369]
[New LWP 175370]
[New LWP 175371]
[New LWP 175372]
[New LWP 175373]
[New LWP 175374]
[New LWP 175375]
[New LWP 175376]
[New LWP 175377]
[New LWP 175378]
[New LWP 175379]
[New LWP 175380]
[New LWP 175381]
[New LWP 175382]
[New LWP 175383]
[New LWP 175384]
[New LWP 175385]
[New LWP 175386]
[New LWP 175387]
[New LWP 175388]
[New LWP 175389]
[New LWP 175390]
[New LWP 175391]
[New LWP 175392]
[New LWP 175403]
[New LWP 175407]
[New LWP 175409]
[New LWP 175414]
[New LWP 175416]
[New LWP 175418]
[New LWP 175419]

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #20 from chris <exxos21@googlemail.com> ---
Created attachment 38576
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38576&action=edit
More G hung

I'm not 100% sure I am doing this trace right, the guy who normally deals with
this stuff is away for the rest of the week :(

I enclose a image of "G" stuck on this. Does seem to close eventually. But
generally takes 400 seconds or more.

Attaching to process 177319
[New LWP 177320]
[New LWP 177321]
[New LWP 177322]
[New LWP 177323]
[New LWP 177324]
[New LWP 177325]
[New LWP 177326]
[New LWP 177327]
[New LWP 177328]
[New LWP 177329]
[New LWP 177330]
[New LWP 177331]
[New LWP 177332]
[New LWP 177333]
[New LWP 177334]
[New LWP 177335]
[New LWP 177336]
[New LWP 177337]
[New LWP 177338]
[New LWP 177339]
[New LWP 177340]
[New LWP 177341]
[New LWP 177342]
[New LWP 177343]
[New LWP 177344]
[New LWP 177346]
[New LWP 177349]
[New LWP 177352]
[New LWP 177354]
[New LWP 177355]
[New LWP 177358]
[New LWP 177360]
[New LWP 177363]
[New LWP 177364]
[New LWP 177365]
[New LWP 177367]
[New LWP 177371]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
__pthread_clockjoin_ex (threadid=140570429146880, thread_return=0x7ffcf5024220,
clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>)
at pthread_join_common.c:145
145 pthread_join_common.c: No such file or directory.
(gdb) c
Continuing.
[Thread 0x7fd921ffb700 (LWP 177355) exited]
[Thread 0x7fd92966a700 (LWP 177349) exited]
[Thread 0x7fd91d7f2700 (LWP 177364) exited]
[Thread 0x7fd9227fc700 (LWP 177354) exited]
[Thread 0x7fd91dff3700 (LWP 177363) exited]
[Thread 0x7fd9207f8700 (LWP 177358) exited]

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #21 from chris <exxos21@googlemail.com> ---
Ages later it chucked out this.



Continuing.
[Thread 0x7fd921ffb700 (LWP 177355) exited]
[Thread 0x7fd92966a700 (LWP 177349) exited]
[Thread 0x7fd91d7f2700 (LWP 177364) exited]
[Thread 0x7fd9227fc700 (LWP 177354) exited]
[Thread 0x7fd91dff3700 (LWP 177363) exited]
[Thread 0x7fd9207f8700 (LWP 177358) exited]
[Thread 0x7fd91bfef700 (LWP 177367) exited]
[Thread 0x7fd91cff1700 (LWP 177365) exited]
[Thread 0x7fd91f7f6700 (LWP 177360) exited]
[Thread 0x7fd91a7ec700 (LWP 177371) exited]
[Thread 0x7fd9237fe700 (LWP 177352) exited]
[Thread 0x7fd92ae6d700 (LWP 177346) exited]
[Thread 0x7fd933eab700 (LWP 177328) exited]
[Thread 0x7fd9346ac700 (LWP 177327) exited]
[Thread 0x7fd92be6f700 (LWP 177344) exited]
[Thread 0x7fd92c674700 (LWP 177343) exited]
[Thread 0x7fd92ce79700 (LWP 177342) exited]
[Thread 0x7fd92de83700 (LWP 177340) exited]
[Thread 0x7fd92e688700 (LWP 177339) exited]
[Thread 0x7fd92ee8d700 (LWP 177338) exited]
[Thread 0x7fd9316a6700 (LWP 177333) exited]
[Thread 0x7fd932ea9700 (LWP 177330) exited]
[Thread 0x7fd9336aa700 (LWP 177329) exited]
[Thread 0x7fd934ead700 (LWP 177326) exited]
[Thread 0x7fd9356ae700 (LWP 177325) exited]
[Thread 0x7fd935eaf700 (LWP 177324) exited]
[Thread 0x7fd9366b0700 (LWP 177323) exited]
[Thread 0x7fd930ea1700 (LWP 177334) exited]
[Thread 0x7fd936eb1700 (LWP 177322) exited]
[Thread 0x7fd9376b2700 (LWP 177321) exited]
[Thread 0x7fd92d67e700 (LWP 177341) exited]
[Thread 0x7fd937eb3700 (LWP 177320) exited]
[Thread 0x7fd92f692700 (LWP 177337) exited]
[Thread 0x7fd92fe97700 (LWP 177336) exited]
[Thread 0x7fd93069c700 (LWP 177335) exited]
[Thread 0x7fd9326a8700 (LWP 177331) exited]
[Thread 0x7fd931ea7700 (LWP 177332) exited]
[Inferior 1 (process 177319) exited normally]

Now looks like all the "G" connections have closed.

Though currently Apache seems to be hanging a lot even though the processor
usage is practically nothing. It seems to be taking a very long time to serve
any pages. I think it gets worse when there is a lot of hung "G" or "W"
connections.

I tried opening up more servers and children But it does not change anything.
There is only generally 5-20 active connections open at any one time. There's
pretty much nothing happening. The requests are only like once a second in the
logs. So not sure What's going on currently.

It is like the original problem is still there, only now it is not using 100%
CPU when things get "stuck".

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #22 from chris <exxos21@googlemail.com> ---
This seems new.


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
__libc_read (nbytes=1, buf=0x7ffe2c8e2473, fd=7) at
../sysdeps/unix/sysv/linux/read.c:26
26 ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) c
Continuing.

Thread 17 "apache2" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fdfb2ffd700 (LWP 181161)]
__GI___writev (iovcnt=1, iov=0x7fdfb83794e0, fd=42) at
../sysdeps/unix/sysv/linux/writev.c:26
26 ../sysdeps/unix/sysv/linux/writev.c: No such file or directory.
(gdb) c
Continuing.

Thread 14 "apache2" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fdfb8d2a700 (LWP 181158)]
__GI___writev (iovcnt=1, iov=0x7fdfb821a4e0, fd=47) at
../sysdeps/unix/sysv/linux/writev.c:26
26 in ../sysdeps/unix/sysv/linux/writev.c
(gdb) c
Continuing.

Thread 20 "apache2" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fdfb17fa700 (LWP 181164)]
__GI___writev (iovcnt=1, iov=0x7fdfb81c84e0, fd=61) at
../sysdeps/unix/sysv/linux/writev.c:26
26 in ../sysdeps/unix/sysv/linux/writev.c
(gdb) c
Continuing.

Thread 19 "apache2" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7fdfb1ffb700 (LWP 181163)]
__GI___writev (iovcnt=5, iov=0x7fdfb80c64e0, fd=29) at
../sysdeps/unix/sysv/linux/writev.c:26
26 in ../sysdeps/unix/sysv/linux/writev.c


I have no idea whats going on now :(

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #23 from chris <exxos21@googlemail.com> ---
ah so I needed to type in backtrace then.

This is one connection which is stuck on "G". Rest of the connections are
inactive.

I did the backtrace a few seconds apart.


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__pthread_clockjoin_ex (threadid=140598715582208, thread_return=0x7ffe2c8e2420,
clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>)
at pthread_join_common.c:145
145 pthread_join_common.c: No such file or directory.
(gdb) backtrace
#0 __pthread_clockjoin_ex (threadid=140598715582208,
thread_return=0x7ffe2c8e2420, clockid=<optimized out>, abstime=<optimized out>,
block=<optimized out>)
at pthread_join_common.c:145
#1 0x00007fdfbff8e3ef in apr_thread_join () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007fdfbfbcdd28 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#3 0x00007fdfbfbcf44b in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#4 0x00007fdfbfbcf7ba in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007fdfbfbd018f in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x0000563afbc78de8 in ap_run_mpm ()
#7 0x0000563afbc707f9 in main ()
(gdb) backtrace
#0 __pthread_clockjoin_ex (threadid=140598715582208,
thread_return=0x7ffe2c8e2420, clockid=<optimized out>, abstime=<optimized out>,
block=<optimized out>)
at pthread_join_common.c:145
#1 0x00007fdfbff8e3ef in apr_thread_join () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007fdfbfbcdd28 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#3 0x00007fdfbfbcf44b in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#4 0x00007fdfbfbcf7ba in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007fdfbfbd018f in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x0000563afbc78de8 in ap_run_mpm ()
#7 0x0000563afbc707f9 in main ()
(gdb) backtrace
#0 __pthread_clockjoin_ex (threadid=140598715582208,
thread_return=0x7ffe2c8e2420, clockid=<optimized out>, abstime=<optimized out>,
block=<optimized out>)
at pthread_join_common.c:145
#1 0x00007fdfbff8e3ef in apr_thread_join () from
target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007fdfbfbcdd28 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#3 0x00007fdfbfbcf44b in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#4 0x00007fdfbfbcf7ba in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007fdfbfbd018f in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x0000563afbc78de8 in ap_run_mpm ()
#7 0x0000563afbc707f9 in main ()

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #24 from Eric Covener <covener@gmail.com> ---
(In reply to chris from comment #23)
> ah so I needed to type in backtrace then.

"thread apply all bt"

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org