Hello all!
I've recently come across an issue that's caused httpd to slowly choke out my server's resources over time. I spent a little bit searching through recently fixed bugs and found this issue<https://bz.apache.org/bugzilla/show_bug.cgi?id=63010> about memory consumption problems with mod_proxy_hcheck, and saw behavior like described in that thread.
I applied the configuration workaround (Setting ProxyHCTPSize to a reasonably large amount so as the threads it needs shouldn't get exhausted even if HCs pile up.) That mostly seemed to clear up the issue, as my resource usage has been considerably lower since then.
However, as I've been keeping an eye on it, I'm still seeing usage creep up. Pulling up my server status page, I noticed the scoreboard full of "L"s, which struck me as incredibly odd. I continued to refresh the status for a while and they just continued to stay stuck. I also noticed it only seemed to happen on workers in the h2 protocol. I've pasted an example of what I'm seeing at the end of this message. These stuck workers are also contributing to an ever-increasing ms/request size. If I give it long enough through periods of low (or no) traffic, they *eventually* seem to time out or finish, but that's not so useful during the workday.
My "workaround" has been a cron job to gracefully restart every so often, but that feels like a huge duct-tape solution.
Does anyone have any idea where I might have gone wrong in my config that I'm ending up with so many workers stuck in this state?
I'm curious, as I've noticed I can't trigger this issue by using HTTP 1.1, if this other issue<https://bz.apache.org/bugzilla/show_bug.cgi?id=64234> is related, but I'm not seeing any messages like speaks of.
Cheers,
Chris
I've recently come across an issue that's caused httpd to slowly choke out my server's resources over time. I spent a little bit searching through recently fixed bugs and found this issue<https://bz.apache.org/bugzilla/show_bug.cgi?id=63010> about memory consumption problems with mod_proxy_hcheck, and saw behavior like described in that thread.
I applied the configuration workaround (Setting ProxyHCTPSize to a reasonably large amount so as the threads it needs shouldn't get exhausted even if HCs pile up.) That mostly seemed to clear up the issue, as my resource usage has been considerably lower since then.
However, as I've been keeping an eye on it, I'm still seeing usage creep up. Pulling up my server status page, I noticed the scoreboard full of "L"s, which struck me as incredibly odd. I continued to refresh the status for a while and they just continued to stay stuck. I also noticed it only seemed to happen on workers in the h2 protocol. I've pasted an example of what I'm seeing at the end of this message. These stuck workers are also contributing to an ever-increasing ms/request size. If I give it long enough through periods of low (or no) traffic, they *eventually* seem to time out or finish, but that's not so useful during the workday.
My "workaround" has been a cron job to gracefully restart every so often, but that feels like a huge duct-tape solution.
Does anyone have any idea where I might have gone wrong in my config that I'm ending up with so many workers stuck in this state?
I'm curious, as I've noticed I can't trigger this issue by using HTTP 1.1, if this other issue<https://bz.apache.org/bugzilla/show_bug.cgi?id=64234> is related, but I'm not seeing any messages like speaks of.
Cheers,
Chris