Mailing List Archive: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.”

Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.”

Mar 29, 2021, 5:45 AM

Post #1 of 15 (489 views)

Hi there,

I'm running some PHP-based sites on CentOS 8, which uses version 2.4.37 of
Apache. Apache is set to use MPM Event mode, which is also the default.

Once per day during a quiet period, logrotate rotates the Apache logs by
doing a graceful restart (reload) as part of a postrotate command:

/bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true

This causes the following line to appear in the Apache error log, which I
believe to be normal and expected:

[mpm_event:notice] [pid 1413:tid 140292271810880] AH00493: SIGUSR1
received. Doing graceful restart

The log is then rotated, and in the newly created error log, I see the
following:

[lbmethod_heartbeat:notice] [pid 1413:tid 140292271810880] AH02282: No
slotmem from mod_heartmonitor
[mpm_event:notice] [pid 1413:tid 140292271810880] AH00489: Apache/2.4.37
(centos) OpenSSL/1.1.1g configured -- resuming normal operations
[core:notice] [pid 1413:tid 140292271810880] AH00094: Command line:
'/usr/sbin/httpd -D FOREGROUND'

Then, immediately after, I see this error repeated once per second:

[mpm_event:error] [pid 1413:tid 140292271810880] AH03490: scoreboard is
full, not at MaxRequestWorkers.Increase ServerLimit.

The server then becomes unresponsive and will not answer HTTPS requests. It
continues logging the above error and will not begin working again until I
manually restart the Apache process.

All configs on this server are essentially the default for CentOS 8. I have
not tried making any adjustments to the MPM Event config settings -- mostly
because I'm not sure what setting for ServerLimit would be "correct," and
also because this problem happens *only* when logrotate does its graceful
restart, and not during other periods of high traffic.

My research on this issue brings up a lot of results about an old Apache
bug related to MPM Event mode, but my version of Apache shouldn't be
affected by this particular bug. I have also seen suggestions that some
long-running processes may hang and eat up all available worker processes,
but I don't understand why this would interrupt the restart. And I don't
have any particularly long-running processes served by Apache -- just
quickly executing PHP scripts.

Can anyone suggest what might fix this issue, given that it seems to happen
only during a graceful reload? Should I try changing the method that
logrotate uses to restart Apache?

Is there some other way I can avoid the server hanging, or is it really
necessary to increase the ServerLimit as the error suggests?

Thank you very much for any insight or help.

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

rammsteinium at me

Mar 29, 2021, 12:52 PM

Post #2 of 15 (488 views)

Permalink

Try this:
https://httpd.apache.org/docs/2.4/programs/rotatelogs.html <https://httpd.apache.org/docs/2.4/programs/rotatelogs.html>

> On 29 Mar 2021, at 13:45, A.J. Gatlin <ajgatlin@gmail.com> wrote:
>
> Hi there,
>
> I'm running some PHP-based sites on CentOS 8, which uses version 2.4.37 of Apache. Apache is set to use MPM Event mode, which is also the default.
>
> Once per day during a quiet period, logrotate rotates the Apache logs by doing a graceful restart (reload) as part of a postrotate command:
>
> /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
>
> This causes the following line to appear in the Apache error log, which I believe to be normal and expected:
>
> [mpm_event:notice] [pid 1413:tid 140292271810880] AH00493: SIGUSR1 received. Doing graceful restart
>
> The log is then rotated, and in the newly created error log, I see the following:
>
> [lbmethod_heartbeat:notice] [pid 1413:tid 140292271810880] AH02282: No slotmem from mod_heartmonitor
> [mpm_event:notice] [pid 1413:tid 140292271810880] AH00489: Apache/2.4.37 (centos) OpenSSL/1.1.1g configured -- resuming normal operations
> [core:notice] [pid 1413:tid 140292271810880] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
>
> Then, immediately after, I see this error repeated once per second:
>
> [mpm_event:error] [pid 1413:tid 140292271810880] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> The server then becomes unresponsive and will not answer HTTPS requests. It continues logging the above error and will not begin working again until I manually restart the Apache process.
>
> All configs on this server are essentially the default for CentOS 8. I have not tried making any adjustments to the MPM Event config settings -- mostly because I'm not sure what setting for ServerLimit would be "correct," and also because this problem happens *only* when logrotate does its graceful restart, and not during other periods of high traffic.
>
> My research on this issue brings up a lot of results about an old Apache bug related to MPM Event mode, but my version of Apache shouldn't be affected by this particular bug. I have also seen suggestions that some long-running processes may hang and eat up all available worker processes, but I don't understand why this would interrupt the restart. And I don't have any particularly long-running processes served by Apache -- just quickly executing PHP scripts.
>
> Can anyone suggest what might fix this issue, given that it seems to happen only during a graceful reload? Should I try changing the method that logrotate uses to restart Apache?
>
> Is there some other way I can avoid the server hanging, or is it really necessary to increase the ServerLimit as the error suggests?
>
> Thank you very much for any insight or help.

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

ajgatlin at gmail

Mar 29, 2021, 4:39 PM

Post #3 of 15 (488 views)

Permalink

On Mon, Mar 29, 2021 at 9:52 PM angel Hall-Coulston
<rammsteinium@me.com.invalid> wrote:

> Try this:
> https://httpd.apache.org/docs/2.4/programs/rotatelogs.html
>

I appreciate the tip, and I believe the use of Apache's own log rotation
would allow me to sidestep the problem.

However, Apache's rotatelogs isn't quite as feature rich as logrotate, and
logrotate has been a de-facto standard on Linux for a very long time, so
I'm really curious about why I can't safely use it to reload Apache as I've
done for the past ~20 years.

Incidentally, I've been able to reproduce the "scoreboard is full" error on
a different CentOS 8 server by using simulated connections and running
"systemctl reload httpd". So it appears that the default settings are not
sufficient to avoid this.

Oddly, I'm not finding this problem widely reported. Do the default
settings for the Event MPM tend to work for most installations?

Thanks.

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

stormy22 at stormy

Mar 29, 2021, 5:11 PM

Post #4 of 15 (488 views)

Permalink

On 2021-03-29 7:39 p.m., A.J. Gatlin wrote:
> On Mon, Mar 29, 2021 at 9:52 PM angel Hall-Coulston
> <rammsteinium@me.com.invalid> wrote:
>
>> Try this:
>> https://httpd.apache.org/docs/2.4/programs/rotatelogs.html
>>
>
> I appreciate the tip, and I believe the use of Apache's own log rotation
> would allow me to sidestep the problem.
>
> However, Apache's rotatelogs isn't quite as feature rich as logrotate, and
> logrotate has been a de-facto standard on Linux for a very long time,

I'm not sure where "rich" becomes "poor"? Been using both for a couple
of decades or more. If it ain't broken, don't fix it. Try "Apache's
rotatelogs" and if you're really unhappy that it needs tweaking,
probably someone here can help you.

Best -- P.
Tired old sys-admin

so
> I'm really curious about why I can't safely use it to reload Apache as I've
> done for the past ~20 years.
>
> Incidentally, I've been able to reproduce the "scoreboard is full" error on
> a different CentOS 8 server by using simulated connections and running
> "systemctl reload httpd". So it appears that the default settings are not
> sufficient to avoid this.
>
> Oddly, I'm not finding this problem widely reported. Do the default
> settings for the Event MPM tend to work for most installations?
>
> Thanks.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

ajgatlin at gmail

Mar 29, 2021, 5:40 PM

Post #5 of 15 (488 views)

Permalink

On Tue, Mar 30, 2021 at 2:11 AM Paul A <stormy22@stormy.ca> wrote:

>
> I'm not sure where "rich" becomes "poor"? Been using both for a couple
> of decades or more. If it ain't broken, don't fix it. Try "Apache's
> rotatelogs" and if you're really unhappy that it needs tweaking,
> probably someone here can help you.
>

Well, I'm not sure if the best solution to "Apache graceful restarts are
causing errors and hangs" is "stop doing Apache graceful restarts." Or is
it?? :-)

I'm sure you're correct that I could work around the problem by changing
the logging method. But I'm not convinced that logrotate is the thing
that's broken. It's just the reason I'm having to reload Apache --
something that, before my migration to Apache 2.4 with Event MPM, was
totally normal and never caused problems.

Are graceful restarts with the Event MPM known to be problematic? Is it
something that can be resolved by tuning the MPM settings? Or should I --
as you seem to suggest -- avoid this type of restart entirely by changing
my logging method?

Thanks very much.

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

jim at netrition

Mar 29, 2021, 5:42 PM

Post #6 of 15 (488 views)

Permalink

On 3/29/2021 8:45 AM, A.J. Gatlin wrote:
> Hi there,
>
> I'm running some PHP-based sites on CentOS 8, which uses version
> 2.4.37 of Apache. Apache is set to use MPM Event mode, which is also
> the default.
>
> Once per day during a quiet period, logrotate rotates the Apache logs
> by doing a graceful restart (reload) as part of a postrotate command:
>
> /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
>
> This causes the following line to appear in the Apache error log,
> which I believe to be normal and expected:
>
> [mpm_event:notice] [pid 1413:tid 140292271810880] AH00493: SIGUSR1
> received. Doing graceful restart
>
> The log is then rotated, and in the newly created error log, I see the
> following:
>
> [lbmethod_heartbeat:notice] [pid 1413:tid 140292271810880] AH02282: No
> slotmem from mod_heartmonitor
> [mpm_event:notice] [pid 1413:tid 140292271810880] AH00489:
> Apache/2.4.37 (centos) OpenSSL/1.1.1g configured -- resuming normal
> operations
> [core:notice] [pid 1413:tid 140292271810880] AH00094: Command line:
> '/usr/sbin/httpd -D FOREGROUND'
>
> Then, immediately after, I see this error repeated once per second:
>
> [mpm_event:error] [pid 1413:tid 140292271810880] AH03490: scoreboard
> is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> The server then becomes unresponsive and will not answer HTTPS
> requests. It continues logging the above error and will not begin
> working again until I manually restart the Apache process.
>
> All configs on this server are essentially the default for CentOS 8. I
> have not tried making any adjustments to the MPM Event config settings
> -- mostly because I'm not sure what setting for ServerLimit would be
> "correct," and also because this problem happens *only* when logrotate
> does its graceful restart, and not during other periods of high traffic.
>
> My research on this issue brings up a lot of results about an old
> Apache bug related to MPM Event mode, but my version of Apache
> shouldn't be affected by this particular bug. I have also seen
> suggestions that some long-running processes may hang and eat up all
> available worker processes, but I don't understand why this would
> interrupt the restart. And I don't have any particularly long-running
> processes served by Apache -- just quickly executing PHP scripts.
>
> Can anyone suggest what might fix this issue, given that it seems to
> happen only during a graceful reload? Should I try changing the method
> that logrotate uses to restart Apache?
>
> Is there some other way I can avoid the server hanging, or is it
> really necessary to increase the ServerLimit as the error suggests?
>
> Thank you very much for any insight or help.

Have you checked your request logs at the time of restart for any
request activity associated with the restart?
Does https://yourserver.com/server-status present any clues,
particularly the processes section or will that also not respond?
Does a restart, graceful or full restart, outside of a logrotate present
the same issue?

Maybe give MaxConnectionsPerChild a setting. I believe the default is 0
which indicates httpd children do not expire. Maybe try 2000 so httpd
children expire occasionally and restart new... just in case of some
memory issue you are facing.
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxconnectionsperchild

Jim

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

covener at gmail

Mar 29, 2021, 5:49 PM

Post #7 of 15 (488 views)

Permalink

> Are graceful restarts with the Event MPM known to be problematic?

Not especially. Not much different from worker. Ultimately if requests
are hung, gracefully exiting processes can hang around a long time and
take up space needed by processes running after the restart.

What are your current MPM settings and how busy is your server during
rotation? If ServerLimit is above MaxClients/ThreadsPerChild the extra
scoreboard space is what allows the old generation of processes to
gracefully exit while a new generation spins up.

Does your logrotate script restart multiple times in succession for
many log files?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

dferradal at apache

Mar 30, 2021, 2:51 AM

Post #8 of 15 (488 views)

Permalink

I may be adding noise, but it is worth checking if you are using third
party modules and which ones.

I don't know which version of Apache worked around issues with some
cases of third party "misbehaving" modules, which would make workers
be gracefully closing for ever and that would also lead to graceful
restart leaving the servers without available workers.

If I were you I would first check the list of modules and with
mod_status the state of workers before issuing a graceful.

Cheers

El mar, 30 mar 2021 a las 2:49, Eric Covener (<covener@gmail.com>) escribió:
>
> > Are graceful restarts with the Event MPM known to be problematic?
>
> Not especially. Not much different from worker. Ultimately if requests
> are hung, gracefully exiting processes can hang around a long time and
> take up space needed by processes running after the restart.
>
> What are your current MPM settings and how busy is your server during
> rotation? If ServerLimit is above MaxClients/ThreadsPerChild the extra
> scoreboard space is what allows the old generation of processes to
> gracefully exit while a new generation spins up.
>
> Does your logrotate script restart multiple times in succession for
> many log files?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>

--
Daniel Ferradal
HTTPD Project
#httpd help at Freenode

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

ajgatlin at gmail

Mar 30, 2021, 5:51 AM

Post #9 of 15 (488 views)

Permalink

On Tue, Mar 30, 2021 at 2:42 AM Jim Albert <jim@netrition.com> wrote:

>
> Have you checked your request logs at the time of restart for any
> request activity associated with the restart?
>

Yes, I did have a look at this to see if there was anything out of the
ordinary. However, I didn't see anything interesting -- no scans or attacks
that would result in especially high numbers of connections. The reload
happens during a relatively quiet overnight period.

> Does https://yourserver.com/server-status present any clues,
> particularly the processes section or will that also not respond?
>

Nothing would respond until I forcibly restarted the httpd process. I was
getting the "scoreboard is full" error logged once per second to the
error_log during this period.

> Does a restart, graceful or full restart, outside of a logrotate present
> the same issue?
>

A full restart always works fine -- no errors. It just takes some time and
is disruptive.
For the graceful restart, I was able to reproduce the issue on a test
machine when I fed it simulated traffic from the tool slowhttptest. I'd
issue the reload command, see the "scoreboard is full" in the logs, and
then the system would be unresponsive as long as my simulated traffic was
continuing. As soon as the simulated traffic ends, the system seems to
recover. It would appear, though, that in production, the system never gets
a chance to recover, because there are always incoming connection requests.
At least that's my guess.

>
> Maybe give MaxConnectionsPerChild a setting. I believe the default is 0
> which indicates httpd children do not expire. Maybe try 2000 so httpd
> children expire occasionally and restart new... just in case of some
> memory issue you are facing.
>
> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxconnectionsperchild

I'll consider it. Thanks very much.

>
>

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

jim at netrition

Mar 30, 2021, 7:58 AM

Post #10 of 15 (488 views)

Permalink

On 3/30/2021 8:51 AM, A.J. Gatlin wrote:
>
>
> On Tue, Mar 30, 2021 at 2:42 AM Jim Albert <jim@netrition.com
> <mailto:jim@netrition.com>> wrote:
>
>
> Have you checked your request logs at the time of restart for any
> request activity associated with the restart?
>
>
> Yes, I did have a look at this to see if there was anything out of the
> ordinary. However, I didn't see anything interesting -- no scans or
> attacks that would result in especially high numbers of connections.
> The reload happens during a relatively quiet overnight period.
>
> Does https://yourserver.com/server-status
> <https://yourserver.com/server-status> present any clues,
> particularly the processes section or will that also not respond?
>
>
> Nothing would respond until I forcibly restarted the httpd process. I
> was getting the "scoreboard is full" error logged once per second to
> the error_log during this period.
>
> Does a restart, graceful or full restart, outside of a logrotate
> present
> the same issue?
>
>
> A full restart always works fine -- no errors. It just takes some time
> and is disruptive.
> For the graceful restart, I was able to reproduce the issue on a test
> machine when I fed it simulated traffic from the tool slowhttptest.
> I'd issue the reload command, see the "scoreboard is full" in the
> logs, and then the system would be unresponsive as long as my
> simulated traffic was continuing. As soon as the simulated traffic
> ends, the system seems to recover. It would appear, though, that in
> production, the system never gets a chance to recover, because there
> are always incoming connection requests. At least that's my guess.
>
>
> Maybe give MaxConnectionsPerChild a setting. I believe the default
> is 0
> which indicates httpd children do not expire. Maybe try 2000 so httpd
> children expire occasionally and restart new... just in case of some
> memory issue you are facing.
> https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxconnectionsperchild
> <https://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxconnectionsperchild>
>
>
> I'll consider it. Thanks very much.
>
>

How about a graceful restart right after a fresh full restart. Does that
immediate graceful restart present the same issue or does the problem
require some period of requests before a graceful restart will present
the problem. Some process information ('top' on UNIX) might tell you if
you have some httpd processes taking a long time to complete. You could
watch your server-status over time to see what might be building up to
the problem state.

Jim

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

ajgatlin at gmail

Mar 30, 2021, 8:20 AM

Post #11 of 15 (488 views)

Permalink

On Tue, Mar 30, 2021 at 2:49 AM Eric Covener <covener@gmail.com> wrote:

> What are your current MPM settings and how busy is your server during
> rotation? If ServerLimit is above MaxClients/ThreadsPerChild the extra
> scoreboard space is what allows the old generation of processes to
> gracefully exit while a new generation spins up.
>

The time during the log rotation is one of the more quiet times of the day.
I didn't specify any MPM settings initially, so everything was the default
for Apache 2.4.37 on CentOS 8, which I believe is as follows:

ServerLimit 16
ThreadsPerChild 25
MaxRequestWorkers 400

I have since manually specified some settings in
/etc/httpd/conf.modules.d/01-cgi.conf. I doubled the ServerLimit to 32 and
the MaxRequestWorkers to 800, so it looks like this:

<IfModule mpm_event_module>
LoadModule cgid_module modules/mod_cgid.so
ServerLimit 32
ThreadsPerChild 25
MaxRequestWorkers 800
</IfModule>

The system has now survived an unattended log rotation and Apache reload
with those settings, but I can't be sure if it's because of the MPM config
change, luck, or something else. It makes me a bit nervous.

Would you suggest that I bump up ServerLimit further, and leave the other
parameters untouched?

> Does your logrotate script restart multiple times in succession for
> many log files?
>

The server runs four VirtualHosts, each with two logs (access and error).
As far as I know, there's only one Apache reload happening after all of
those are rotated.

Thanks!

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

ajgatlin at gmail

Mar 30, 2021, 8:29 AM

Post #12 of 15 (488 views)

Permalink

On Tue, Mar 30, 2021 at 11:52 AM Daniel Ferradal <dferradal@apache.org>
wrote:

> I may be adding noise, but it is worth checking if you are using third
> party modules and which ones.
>

Thanks for the tip. I'm not using any third party modules. I had a hunch
that maybe mod_security was causing issues, but on my testbed I could
reproduce the issue regardless of whether mod_security was being used.

> If I were you I would first check the list of modules and with
> mod_status the state of workers before issuing a graceful.
>

I had a look at this on my testbed. I tried to send about 50 requests per
second with the slowhttptest utility, which caused all worker processes to
be used up. When I issued a graceful reload, I could see some of those
processes enter the "G" state. After that, I was unable to refresh the
server stats, because all connections were used. Eventually, however, the
system came back to life again. I don't know if that's because slowhttptest
was backing off, or if things were just working as intended. On my
production system, that didn't happen: it sat for over 2 hours in an
unresponsive state after the graceful restart. Perhaps that's because it
was receiving constant connection attempts, and these were occupying all
available connections and preventing the reload from finishing?

I'm at a loss regarding what else might have been misbehaving to hold onto
the connections. Everything was working fine up until the reload.

Thanks!

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

covener at gmail

Mar 30, 2021, 9:34 AM

Post #13 of 15 (488 views)

Permalink

> <IfModule mpm_event_module>
> LoadModule cgid_module modules/mod_cgid.so
> ServerLimit 32
> ThreadsPerChild 25
> MaxRequestWorkers 800
> </IfModule>
>
> The system has now survived an unattended log rotation and Apache reload with those settings, but I can't be sure if it's because of the MPM config change, luck, or something else. It makes me a bit nervous.
>
> Would you suggest that I bump up ServerLimit further, and leave the other parameters untouched?

With double the capacity now it might never matter -- but in general
yes you'd want to have "slack space" with ServerLimit of e.g. 48 or
64. The slack space is occupied by replacements when termination of
the old processes is slow

It would also be good to find out what slows down the processes to
exit, this is where the workload/config comes into play. pstack or
gdb will give you a backtrace of processes. If server-status is
reachable you can investigate the 'G' processes. If it's not
available, you can use ps to look for processes that have an unusual #
of threads (the threads that were idle terminate, the spinning/hanging
ones are left)

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

ajgatlin at gmail

Mar 30, 2021, 4:49 PM

Post #14 of 15 (476 views)

Permalink

On Tue, Mar 30, 2021 at 4:58 PM Jim Albert <jim@netrition.com> wrote:

>
> How about a graceful restart right after a fresh full restart. Does that
> immediate graceful restart present the same issue or does the problem
> require some period of requests before a graceful restart will present the
> problem. Some process information ('top' on UNIX) might tell you if you
> have some httpd processes taking a long time to complete. You could watch
> your server-status over time to see what might be building up to the
> problem state.
>

I've tried this on my testbed, but it's difficult to replicate realistic
traffic with a mix of users, bots, probe attempts blocked by mod_security,
etc. However, during my tests I haven't noticed any considerable
long-running processes building up, so the full restart/graceful restart
scenario you mention doesn't appear to be any different than letting the
system run for a while. The problem seems to occur when 1) a graceful
restart is called, 2) the system is getting some connection attempts, and
3) the MPM settings aren't sufficient to handle the incoming requests while
waiting for old ones to close.

For now, I'm trying to solve the problem by increasing some MPM parameters,
but it's all a bit nebulous, since I'll never be completely sure that the
values are sufficient to prevent another deadlock situation in which the
server just hangs.

Thanks a lot for your tips.

>

Re: Apache 2.4.37 graceful restart causes error “scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.” [ In reply to ]

ylavic.dev at gmail

Mar 31, 2021, 2:32 AM

Post #15 of 15 (476 views)

Permalink

On Wed, Mar 31, 2021 at 1:49 AM A.J. Gatlin <ajgatlin@gmail.com> wrote:
>
> For now, I'm trying to solve the problem by increasing some MPM parameters, but it's all a bit nebulous, since I'll never be completely sure that the values are sufficient to prevent another deadlock situation in which the server just hangs.

I usually configure something like this, based on the maximum number
of simultaneous clients/workers needed (numWorkers).

# Some pre-computations
numWorkers = [your needs]
if numWorkers < 1000
numProcesses = 10
elif numWorkers < 10000
numProcesses = numWorkers / 100
else
numProcesses = 100
endif
numThreads = numWorkers / numProcesses

# MPM settings
StartServers 1
ServerLimit $numProcesses * 2.5
ThreadLimit $numThreads
ThreadsPerChild $numThreads
MinSpareThreads $numThreads
MaxSpareThreads $numWorkers / 2
MaxRequestWorkers $numWorkers
MaxConnectionsPerChild 0

Hth..

Regards;
Yann.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org