Mailing List Archive

1 2  View All
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #25 from chris <exxos21@googlemail.com> ---
Created attachment 38577
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38577&action=edit
Single G hung

Thanks.

The image is what is shown in the status as the current hung "G" connection.

This is the other info..


(gdb) thread apply all bt

Thread 13 (Thread 0x7fdfb07f8700 (LWP 185185)):
#0 0x00007fdfbfe6b46e in epoll_wait (epfd=18, events=0x7fdfbfc01488,
maxevents=23, timeout=239742) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007fdfbff8998e in ?? () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007fdfbfbd17f8 in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#3 0x00007fdfbff46609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#4 0x00007fdfbfe6b133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7fdfb37fe700 (LWP 185179)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7fdfbfc01160) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fdfbfc010e8,
cond=0x7fdfbfc01138) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7fdfbfc01138, mutex=0x7fdfbfc010e8) at
pthread_cond_wait.c:647
#3 0x0000563afbca746d in ap_queue_pop_something ()
#4 0x00007fdfbfbd26fe in ?? () from
target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007fdfbff46609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#6 0x00007fdfbfe6b133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7fdfbad2e700 (LWP 185173)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7fdfbfbb9b34) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fdfbfbb91c8,
cond=0x7fdfbfbb9b08) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7fdfbfbb9b08, mutex=0x7fdfbfbb91c8) at
pthread_cond_wait.c:647
#3 0x00007fdfbf8966f1 in slot_run (thread=0x7fdfbf7f80b0, wctx=0x7fdfbfbb95d8)
at h2_workers.c:340
#4 0x00007fdfbff46609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007fdfbfe6b133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fdfbb52f700 (LWP 185172)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7fdfbfbb9adc) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fdfbfbb91c8,
cond=0x7fdfbfbb9ab0) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7fdfbfbb9ab0, mutex=0x7fdfbfbb91c8) at
pthread_cond_wait.c:647
#3 0x00007fdfbf8966f1 in slot_run (thread=0x7fdfbf7fc0b0, wctx=0x7fdfbfbb9580)
at h2_workers.c:340
#4 0x00007fdfbff46609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007fdfbfe6b133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fdfbbd30700 (LWP 185171)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7fdfbfbb9a84) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fdfbfbb91c8,
cond=0x7fdfbfbb9a58) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x7fdfbfbb9a58, mutex=0x7fdfbfbb91c8) at
pthread_cond_wait.c:647
#3 0x00007fdfbf8966f1 in slot_run (thread=0x7fdfbf8000b0, wctx=0x7fdfbfbb9528)
at h2_workers.c:340
#4 0x00007fdfbff46609 in start_thread (arg=<optimized out>) at
pthread_create.c:477
#5 0x00007fdfbfe6b133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fdfbc531700 (LWP 185170)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0,
futex_word=0x7fdfbfbb9a2c) at ../sysdeps/nptl/futex-internal.h:183
--Type <RET> for more, q to quit, c to continue without paging--

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #26 from chris <exxos21@googlemail.com> ---
Isn't this bug report the same thing ?
https://bz.apache.org/bugzilla/show_bug.cgi?id=65180

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #27 from chris <exxos21@googlemail.com> ---
I was just running some benchmarks between EVENT and PREFORK.


(EVENT)
Percentage of the requests served within a certain time (ms)
50% 6936
66% 7376
75% 7726
80% 7904
90% 8548
95% 8972
98% 9320
99% 9536
100% 11648 (longest request)


(PREFORK)
Percentage of the requests served within a certain time (ms)
50% 4466
66% 4624
75% 4724
80% 4789
90% 5152
95% 5504
98% 5816
99% 5906
100% 6425 (longest request)


I've had to leave it in prefork as several of my forum members are saying they
are constantly getting gateway timeout and then everything works normally for a
few moments and then goes back to gateway timeout again.

Though it is pretty bizarre because yesterday I was hammering the server with
benchmarks and it seemed to cope fine. They don't do anything overnight and
come back the next day and everything has gone crazy slow again.

Restarting Apache seems to make things run incredibly quick for a certain
period of time, possibly minutes to hours and then it just seems to
progressively gets slower and slower as time goes on.

It is interesting that the benchmarks seem to be getting slower a lot worse
with EVENT than PREFORK. I thought EVENT was supposed to be more efficient ?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #28 from chris <exxos21@googlemail.com> ---
Been running prefork fine for the past couple of days. So switch backed to
event just, and immediately start getting gateway timeout errors and everything
was running incredibly slow again.

The test I'm doing now is event without HTTP2 module enabled. So far this isn't
coming out with gateway timeout errors and seems be running reasonably well
currently.

I did some benchmarks for it as well.. event seems to be running slightly
quicker than my previous tests. Though with HTTP2 disabled it seems to be
running marginally slower than prefork.


PREFORK
Percentage of the requests served within a certain time (ms)
50% 4616
66% 4796
75% 4888
80% 4944
90% 5088
95% 5248
98% 5432
99% 5532
100% 6957 (longest request)


EVENT
Percentage of the requests served within a certain time (ms)
50% 4327
66% 4448
75% 4536
80% 4587
90% 4708
95% 4782
98% 4896
99% 4936
100% 5375 (longest request)



EVENT WITHOUT HTTP2
Percentage of the requests served within a certain time (ms)
50% 5120
66% 5320
75% 5460
80% 5576
90% 5816
95% 6032
98% 6300
99% 6396
100% 6608 (longest request)


I'm not sure what else I can realistically test here. Though currently it
seems be only relating to the HTTP2 module rather than event / prefork.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #29 from chris <exxos21@googlemail.com> ---
Created attachment 38581
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38581&action=edit
mod filesize different

Isn't this odd ? the one on the right (new server but same distro) is a much
lower file size than the one on the left (current server).

But more to the point why has that file been accessed recently ?!

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #30 from Eric Covener <covener@gmail.com> ---
(In reply to chris from comment #29)
> Created attachment 38581 [details]
> mod filesize different
>
> Isn't this odd ? the one on the right (new server but same distro) is a
> much lower file size than the one on the left (current server).
>
> But more to the point why has that file been accessed recently ?!

did you overlay one w/ patched version built outside of the distro? Size could
be debug symbols?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #31 from chris <exxos21@googlemail.com> ---
(In reply to Eric Covener from comment #30)

> did you overlay one w/ patched version built outside of the distro? Size
> could be debug symbols?

oh yep, that be the one I built won't it. Not sure how it was compiled because
it is the first time I compiled anything. But if it does debug information then
probably why it is larger.

Just slightly concerned for a moment that it could have been hacked
externally.. My bad there then :)

Is there any ideas why HTTP2 would be failing ?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #32 from Pascal <pascal.christen@hostpoint.ch> ---
(In reply to Pascal from comment #15)
> We also have problems like this (since Apache 2.4.55 - FreeBSD) where
> afterwards processes are in the "uwait" state and are using 100% CPU. I'm
> currently testing the patch you posted and will give you feedback by the end
> of the week.

One of the systems is running the patch and no processes with uwait/100% CPU
have occurred on this system since Tuesday (usually happened every day). From
what I can see this patch should also work for the bundled version of mod_http2
for Apache 2.4.57. Stefan, do you think this patch is safe to apply on Apache
2.4.57 with the bundled mod_http2 version?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #33 from Stefan Eissing <stefan@eissing.org> ---
Yes, this patch should be safe on the last released version.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #34 from Stefan Eissing <stefan@eissing.org> ---
Fix added to trunk as r1910331.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 66624] Using 100% CPU [ In reply to ]
https://bz.apache.org/bugzilla/show_bug.cgi?id=66624

--- Comment #35 from Pascal <pascal.christen@hostpoint.ch> ---
(In reply to Stefan Eissing from comment #34)
> Fix added to trunk as r1910331.

Thank you very much! I'm wondering if you know if there are some special
requests that can trigger the behaviour, or is it ultimately just the amount of
missing cleanups that is causing this behaviour?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org

1 2  View All