Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Apache>
  3. Users
Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit
miked at dewhirst
Nov 22, 2022, 3:14 PM
Post #1 of 14 (1017 views)
Permalink
Can anyone suggest some mpm_event.conf settings for a lightly loaded
DigitalOcean Ubuntu 2022.04 LTS droplet with 8GB of RAM and a "dedicated
CPU"?

I just had to restart Apache which crashed around 22:22 UTC. There is
8Mb of the above error in the log.

This is the plain Apache2 error.log rather than the named website error
log which only shows expected errors such as reCaptcha errors.

It seems to have started filling the scoreboard 24 minutes after
logrotate. The first ten log entries after logrotate are ...

[Tue Nov 22 00:00:01.934987 2022] [mpm_event:notice] [pid 49892:tid
140450581641088] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Tue Nov 22 00:00:01.935032 2022] [core:notice] [pid 49892:tid
140450581641088] AH00094: Command line: '/usr/sbin/apache2'
[Tue Nov 22 00:20:13.411015 2022] [mpm_event:notice] [pid 49892:tid
140450581641088] AH00493: SIGUSR1 received.  Doing graceful restart
[Tue Nov 22 00:20:13.514771 2022] [mpm_event:notice] [pid 49892:tid
140450581641088] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Tue Nov 22 00:20:13.514804 2022] [core:notice] [pid 49892:tid
140450581641088] AH00094: Command line: '/usr/sbin/apache2'
[Tue Nov 22 00:23:57.862865 2022] [mpm_event:notice] [pid 49892:tid
140450581641088] AH00493: SIGUSR1 received.  Doing graceful restart
[Tue Nov 22 00:23:57.937161 2022] [mpm_event:notice] [pid 49892:tid
140450581641088] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Tue Nov 22 00:23:57.937192 2022] [core:notice] [pid 49892:tid
140450581641088] AH00094: Command line: '/usr/sbin/apache2'
[Tue Nov 22 00:23:59.938800 2022] [mpm_event:error] [pid 49892:tid
140450581641088] AH03490: scoreboard is full, not at
MaxRequestWorkers.Increase ServerLimit.
[Tue Nov 22 00:24:00.939954 2022] [mpm_event:error] [pid 49892:tid
140450581641088] AH03490: scoreboard is full, not at
MaxRequestWorkers.Increase ServerLimit.

Then later ...

[Tue Nov 22 22:28:52.526681 2022] [mpm_event:error] [pid 111175:tid
140011041032064] AH03490: scoreboard is full, not at
MaxRequestWorkers.Increase ServerLimit.
[Tue Nov 22 22:28:52.526899 2022] [mpm_event:notice] [pid 111175:tid
140011041032064] AH00492: caught SIGWINCH, shutting down gracefully
[Tue Nov 22 22:28:55.888384 2022] [core:warn] [pid 111175:tid
140011041032064] AH00045: child process 113420 still did not exit,
sending a SIGTERM

... more of the same ...

[Tue Nov 22 22:28:59.893773 2022] [core:warn] [pid 111175:tid
140011041032064] AH00045: child process 113426 still did not exit,
sending a SIGTERM
[Tue Nov 22 22:29:01.896130 2022] [core:error] [pid 111175:tid
140011041032064] AH00046: child process 113420 still did not exit,
sending a SIGKILL

... more of the same ...

[Tue Nov 22 22:29:01.896389 2022] [core:error] [pid 111175:tid
140011041032064] AH00046: child process 113426 still did not exit,
sending a SIGKILL

... I assume this is where I said restart ...

[Tue Nov 22 22:29:03.055559 2022] [mpm_event:notice] [pid 122709:tid
139966049515392] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Tue Nov 22 22:29:03.055753 2022] [core:notice] [pid 122709:tid
139966049515392] AH00094: Command line: '/usr/sbin/apache2'

... and no errors since.

Below is the server-status after restart and below that is my current
mpm_event.conf

Many thanks

Mike

Server Version: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0
Python/3.10

Server MPM: event
Server Built: 2022-09-30T04:09:50

------------------------------------------------------------------------

Current Time: Tuesday, 22-Nov-2022 22:41:35 UTC
Restart Time: Tuesday, 22-Nov-2022 22:29:03 UTC
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 12 minutes 32 seconds
Server load: 0.00 0.00 0.00
Total accesses: 14 - Total Traffic: 121 kB - Total Duration: 10274
CPU Usage: u10.91 s.38 cu0 cs0 - 1.5% CPU load
.0186 requests/sec - 164 B/second - 8.6 kB/request - 733.857 ms/request
1 requests currently being processed, 449 idle workers

Slot PID Stopping Connections Threads Async connections
total accepting busy idle writing keep-alive closing
0 122713 no 0 yes 0 50 0 0 0
1 122714 no 0 yes 0 50 0 0 0
2 122715 no 0 yes 0 50 0 0 0
3 122716 no 0 yes 0 50 0 0 0
4 122717 no 0 yes 0 50 0 0 0
5 123029 no 0 yes 0 50 0 0 0
6 123082 no 0 yes 0 50 0 0 0
7 123083 no 0 yes 1 49 0 0 0
8 123190 no 0 yes 0 50 0 0 0
Sum 9 0 0 1 449 0 0 0



# event MPM
# ServerLimit: Upper limit on configurable number of processes (default 16)
# StartServers: initial number of server processes to start (default 3)
# MinSpareThreads: minimum number of worker threads which are kept spare
(default 75)
# MaxSpareThreads: maximum number of worker threads which are kept spare
(default 250)
# ThreadLimit: upper limit on the configurable number of threads per
child process (default 64)
# ThreadsPerChild: constant number of worker threads in each server
process (default 25)
# MaxRequestWorkers: maximum number of worker threads
(ServerLimit*ThreadsPerChild)
# MaxConnectionsPerChild: maximum number of requests a server process serves
##
# After reading
https://www.liquidweb.com/kb/apache-performance-tuning-apache-mpm-modules/
<IfModule mpm_event_module>
        ServerLimit                     16
        StartServers                    16
        MinSpareThreads                 400
        MaxSpareThreads                 250
        ThreadLimit                     64
        ThreadsPerChild                 50
        AsyncRequestWorkerFactor        2
        MaxRequestWorkers               800
        MaxConnectionsPerChild          0
</IfModule>



--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attached Files:

Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
covener at gmail
Nov 22, 2022, 3:31 PM
Post #2 of 14 (1017 views)
Permalink
On Tue, Nov 22, 2022 at 6:15 PM Mike Dewhirst <miked@dewhirst.com.au> wrote:

> Can anyone suggest some mpm_event.conf settings for a lightly loaded
> DigitalOcean Ubuntu 2022.04 LTS droplet with 8GB of RAM and a "dedicated
> CPU"?
>
> I just had to restart Apache which crashed around 22:22 UTC. There is 8Mb
> of the above error in the log.
>
> This is the plain Apache2 error.log rather than the named website error
> log which only shows expected errors such as reCaptcha errors.
>
> It seems to have started filling the scoreboard 24 minutes after
> logrotate. The first ten log entries after logrotate are ...
>
> [Tue Nov 22 00:00:01.934987 2022] [mpm_event:notice] [pid 49892:tid
> 140450581641088] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
> mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Tue Nov 22 00:00:01.935032 2022] [core:notice] [pid 49892:tid
> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
> [Tue Nov 22 00:20:13.411015 2022] [mpm_event:notice] [pid 49892:tid
> 140450581641088] AH00493: SIGUSR1 received. Doing graceful restart
> [Tue Nov 22 00:20:13.514771 2022] [mpm_event:notice] [pid 49892:tid
> 140450581641088] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
> mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Tue Nov 22 00:20:13.514804 2022] [core:notice] [pid 49892:tid
> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
> [Tue Nov 22 00:23:57.862865 2022] [mpm_event:notice] [pid 49892:tid
> 140450581641088] AH00493: SIGUSR1 received. Doing graceful restart
> [Tue Nov 22 00:23:57.937161 2022] [mpm_event:notice] [pid 49892:tid
> 140450581641088] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
> mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Tue Nov 22 00:23:57.937192 2022] [core:notice] [pid 49892:tid
> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
> [Tue Nov 22 00:23:59.938800 2022] [mpm_event:error] [pid 49892:tid
> 140450581641088] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
> [Tue Nov 22 00:24:00.939954 2022] [mpm_event:error] [pid 49892:tid
> 140450581641088] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
>

With event, ServerLimit should be larger than
MaxRequestWorkers/ThreadsPerChild not equal to it. It only matters when
processes exit and are replaced.
This allows slow to exit processes to be managed alongside their
replacement processes.

...

<IfModule mpm_event_module>
> ServerLimit 16
>
StartServers 16
> MinSpareThreads 400
> MaxSpareThreads 250
> ThreadLimit 64
> ThreadsPerChild 50
> AsyncRequestWorkerFactor 2
> MaxRequestWorkers 800
> MaxConnectionsPerChild 0
> </IfModule>
>

I would start with ServerLimit 24 or 32. Nothing has to change along with
it.
Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
miked at dewhirst
Nov 22, 2022, 3:40 PM
Post #3 of 14 (1017 views)
Permalink
On 23/11/2022 10:31 am, Eric Covener wrote:
>
>
> On Tue, Nov 22, 2022 at 6:15 PM Mike Dewhirst <miked@dewhirst.com.au>
> wrote:
>
> Can anyone suggest some mpm_event.conf settings for a lightly
> loaded DigitalOcean Ubuntu 2022.04 LTS droplet with 8GB of RAM and
> a "dedicated CPU"?
>
> I just had to restart Apache which crashed around 22:22 UTC. There
> is 8Mb of the above error in the log.
>
> This is the plain Apache2 error.log rather than the named website
> error log which only shows expected errors such as reCaptcha errors.
>
> It seems to have started filling the scoreboard 24 minutes after
> logrotate. The first ten log entries after logrotate are ...
>
> [Tue Nov 22 00:00:01.934987 2022] [mpm_event:notice] [pid
> 49892:tid 140450581641088] AH00489: Apache/2.4.52 (Ubuntu)
> OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming
> normal operations
> [Tue Nov 22 00:00:01.935032 2022] [core:notice] [pid 49892:tid
> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
> [Tue Nov 22 00:20:13.411015 2022] [mpm_event:notice] [pid
> 49892:tid 140450581641088] AH00493: SIGUSR1 received. Doing
> graceful restart
> [Tue Nov 22 00:20:13.514771 2022] [mpm_event:notice] [pid
> 49892:tid 140450581641088] AH00489: Apache/2.4.52 (Ubuntu)
> OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming
> normal operations
> [Tue Nov 22 00:20:13.514804 2022] [core:notice] [pid 49892:tid
> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
> [Tue Nov 22 00:23:57.862865 2022] [mpm_event:notice] [pid
> 49892:tid 140450581641088] AH00493: SIGUSR1 received. Doing
> graceful restart
> [Tue Nov 22 00:23:57.937161 2022] [mpm_event:notice] [pid
> 49892:tid 140450581641088] AH00489: Apache/2.4.52 (Ubuntu)
> OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming
> normal operations
> [Tue Nov 22 00:23:57.937192 2022] [core:notice] [pid 49892:tid
> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
> [Tue Nov 22 00:23:59.938800 2022] [mpm_event:error] [pid 49892:tid
> 140450581641088] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
> [Tue Nov 22 00:24:00.939954 2022] [mpm_event:error] [pid 49892:tid
> 140450581641088] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
>
>
> With event, ServerLimit should be larger than
> MaxRequestWorkers/ThreadsPerChild not equal to it. It only matters
> when processes exit and are replaced.
> This allows slow to exit processes to be managed alongside their
> replacement processes.
>
> ...
>
> <IfModule mpm_event_module>
> ServerLimit 16
>
>         StartServers                    16
>         MinSpareThreads                 400
>         MaxSpareThreads                 250
>         ThreadLimit                     64
>         ThreadsPerChild                 50
>         AsyncRequestWorkerFactor        2
>         MaxRequestWorkers               800
>         MaxConnectionsPerChild          0
> </IfModule>
>
>
> I would start with ServerLimit 24 or 32. Nothing has to change along
> with it.

Many thanks Eric. 32 it is!

Cheers

Mike



--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attached Files:

Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
miked at dewhirst
Nov 23, 2022, 4:32 PM
Post #4 of 14 (1012 views)
Permalink
On 23/11/2022 10:40 am, Mike Dewhirst wrote:
> On 23/11/2022 10:31 am, Eric Covener wrote:
>>
>>
>> On Tue, Nov 22, 2022 at 6:15 PM Mike Dewhirst <miked@dewhirst.com.au>
>> wrote:
>>
>> Can anyone suggest some mpm_event.conf settings for a lightly
>> loaded DigitalOcean Ubuntu 2022.04 LTS droplet with 8GB of RAM
>> and a "dedicated CPU"?
>>
>> I just had to restart Apache which crashed around 22:22 UTC.
>> There is 8Mb of the above error in the log.
>>
>> This is the plain Apache2 error.log rather than the named website
>> error log which only shows expected errors such as reCaptcha errors.
>>
>> It seems to have started filling the scoreboard 24 minutes after
>> logrotate. The first ten log entries after logrotate are ...
>>
>> [Tue Nov 22 00:00:01.934987 2022] [mpm_event:notice] [pid
>> 49892:tid 140450581641088] AH00489: Apache/2.4.52 (Ubuntu)
>> OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming
>> normal operations
>> [Tue Nov 22 00:00:01.935032 2022] [core:notice] [pid 49892:tid
>> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
>> [Tue Nov 22 00:20:13.411015 2022] [mpm_event:notice] [pid
>> 49892:tid 140450581641088] AH00493: SIGUSR1 received.  Doing
>> graceful restart
>> [Tue Nov 22 00:20:13.514771 2022] [mpm_event:notice] [pid
>> 49892:tid 140450581641088] AH00489: Apache/2.4.52 (Ubuntu)
>> OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming
>> normal operations
>> [Tue Nov 22 00:20:13.514804 2022] [core:notice] [pid 49892:tid
>> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
>> [Tue Nov 22 00:23:57.862865 2022] [mpm_event:notice] [pid
>> 49892:tid 140450581641088] AH00493: SIGUSR1 received.  Doing
>> graceful restart
>> [Tue Nov 22 00:23:57.937161 2022] [mpm_event:notice] [pid
>> 49892:tid 140450581641088] AH00489: Apache/2.4.52 (Ubuntu)
>> OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming
>> normal operations
>> [Tue Nov 22 00:23:57.937192 2022] [core:notice] [pid 49892:tid
>> 140450581641088] AH00094: Command line: '/usr/sbin/apache2'
>> [Tue Nov 22 00:23:59.938800 2022] [mpm_event:error] [pid
>> 49892:tid 140450581641088] AH03490: scoreboard is full, not at
>> MaxRequestWorkers.Increase ServerLimit.
>> [Tue Nov 22 00:24:00.939954 2022] [mpm_event:error] [pid
>> 49892:tid 140450581641088] AH03490: scoreboard is full, not at
>> MaxRequestWorkers.Increase ServerLimit.
>>
>>
>> With event, ServerLimit should be larger than
>> MaxRequestWorkers/ThreadsPerChild not equal to it. It only matters
>> when processes exit and are replaced.
>> This allows slow to exit processes to be managed alongside their
>> replacement processes.
>>
>> ...
>>
>> <IfModule mpm_event_module>
>> ServerLimit 16
>>
>>         StartServers                    16
>>         MinSpareThreads                 400
>>         MaxSpareThreads                 250
>>         ThreadLimit                     64
>>         ThreadsPerChild                 50
>>         AsyncRequestWorkerFactor        2
>>         MaxRequestWorkers               800
>>         MaxConnectionsPerChild          0
>> </IfModule>
>>
>>
>> I would start with ServerLimit 24 or 32. Nothing has to change along
>> with it.
>
> Many thanks Eric. 32 it is!

Just crashed again.

Should I adjust /etc/logrotate.d/apache2 to restart instead of reload?

Looks like a simple interim repair.

Thanks

Mike

>
> Cheers
>
> Mike
>
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.


--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attached Files:

Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
covener at gmail
Nov 23, 2022, 5:13 PM
Post #5 of 14 (1012 views)
Permalink
> Should I adjust /etc/logrotate.d/apache2 to restart instead of reload?

How many entries for Apache logs are there, and how frequently do they
rotate in logrotated?

I think there's a trick to batch them if there's very many. But it
sounds like it's constantly doing rotation.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
miked at dewhirst
Nov 23, 2022, 5:37 PM
Post #6 of 14 (1012 views)
Permalink
On 24/11/2022 12:13 pm, Eric Covener wrote:
>> Should I adjust /etc/logrotate.d/apache2 to restart instead of reload?
> How many entries for Apache logs are there, and how frequently do they
> rotate in logrotated?

There are three Apache website logs and they rotate daily. Here is the
script

# /etc/logrotate.d/apache2

/var/log/apache2/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0660 root adm
    sharedscripts
    prerotate
    if [ -d /etc/logrotate.d/httpd-prerotate ]; then
        run-parts /etc/logrotate.d/httpd-prerotate
    fi
    endscript
    postrotate
    if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
        invoke-rc.d apache2 reload 2>&1 | logger -t apache2.logrotate
    fi
    endscript
}

# /etc/cron.daily/apache2

#!/bin/sh

# run htcacheclean if set to 'cron' mode

set -e
set -u

type htcacheclean > /dev/null 2>&1 || exit 0
[ -e /etc/default/apache-htcacheclean ] || exit 0


# edit /etc/default/apache-htcacheclean to change this
HTCACHECLEAN_MODE=daemon
HTCACHECLEAN_RUN=auto
HTCACHECLEAN_SIZE=300M
HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk
HTCACHECLEAN_OPTIONS=""

. /etc/default/apache-htcacheclean

[ "$HTCACHECLEAN_MODE" = "cron" ] || exit 0

htcacheclean ${HTCACHECLEAN_OPTIONS}    \
        -p${HTCACHECLEAN_PATH}    \
        -l${HTCACHECLEAN_SIZE}



>
> I think there's a trick to batch them if there's very many. But it
> sounds like it's constantly doing rotation.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:users-unsubscribe@httpd.apache.org
> For additional commands, e-mail:users-help@httpd.apache.org
>


--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attached Files:

Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
dferradal at apache
Nov 23, 2022, 11:48 PM
Post #7 of 14 (1012 views)
Permalink
Some script is probably hogging the workers and not letting them close so
if you must, just restart, but judging by the amount of traffic you seem to
have at the time I would truncate in logrotate and not restart or reload
the server at all.

El jue, 24 nov 2022 2:38, Mike Dewhirst <miked@dewhirst.com.au> escribió:

> On 24/11/2022 12:13 pm, Eric Covener wrote:
>
> Should I adjust /etc/logrotate.d/apache2 to restart instead of reload?
>
> How many entries for Apache logs are there, and how frequently do they
> rotate in logrotated?
>
>
> There are three Apache website logs and they rotate daily. Here is the
> script
>
> # /etc/logrotate.d/apache2
>
> /var/log/apache2/*.log {
> daily
> missingok
> rotate 14
> compress
> delaycompress
> notifempty
> create 0660 root adm
> sharedscripts
> prerotate
> if [ -d /etc/logrotate.d/httpd-prerotate ]; then
> run-parts /etc/logrotate.d/httpd-prerotate
> fi
> endscript
> postrotate
> if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
> invoke-rc.d apache2 reload 2>&1 | logger -t apache2.logrotate
> fi
> endscript
> }
>
> # /etc/cron.daily/apache2
>
> #!/bin/sh
>
> # run htcacheclean if set to 'cron' mode
>
> set -e
> set -u
>
> type htcacheclean > /dev/null 2>&1 || exit 0
> [ -e /etc/default/apache-htcacheclean ] || exit 0
>
>
> # edit /etc/default/apache-htcacheclean to change this
> HTCACHECLEAN_MODE=daemon
> HTCACHECLEAN_RUN=auto
> HTCACHECLEAN_SIZE=300M
> HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk
> HTCACHECLEAN_OPTIONS=""
>
> . /etc/default/apache-htcacheclean
>
> [ "$HTCACHECLEAN_MODE" = "cron" ] || exit 0
>
> htcacheclean ${HTCACHECLEAN_OPTIONS} \
> -p${HTCACHECLEAN_PATH} \
> -l${HTCACHECLEAN_SIZE}
>
>
>
> I think there's a trick to batch them if there's very many. But it
> sounds like it's constantly doing rotation.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
miked at dewhirst
Nov 24, 2022, 3:53 AM
Post #8 of 14 (1005 views)
Permalink
Daniel - thanks for that. Just looked at copytruncate and that seems a good strategy. I'll restart Apache in the morning and try the truncate option over the weekend when I can keep an eye on it.CheersMike--(Unsigned mail from my phone)
-------- Original message --------From: Daniel Ferradal <dferradal@apache.org> Date: 24/11/22 18:48 (GMT+10:00) To: "<users@httpd.apache.org>" <users@httpd.apache.org> Subject: Re: [users@httpd] Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit Some script is probably hogging the workers and not letting them close so if you must, just restart, but judging by the amount of traffic you seem to have at the time I would truncate in logrotate and not restart or reload the server at all.El jue, 24 nov 2022 2:38, Mike Dewhirst <miked@dewhirst.com.au> escribió:




On 24/11/2022 12:13 pm, Eric Covener
wrote:



Should I adjust /etc/logrotate.d/apache2 to restart instead of reload?


How many entries for Apache logs are there, and how frequently do they
rotate in logrotated?


There are three Apache website logs and they rotate daily. Here is
the script

# /etc/logrotate.d/apache2

/var/log/apache2/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0660 root adm
    sharedscripts
    prerotate
    if [ -d /etc/logrotate.d/httpd-prerotate ]; then
        run-parts /etc/logrotate.d/httpd-prerotate
    fi
    endscript
    postrotate
    if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
        invoke-rc.d apache2 reload 2>&1 | logger -t
apache2.logrotate
    fi
    endscript
}

# /etc/cron.daily/apache2

#!/bin/sh

# run htcacheclean if set to 'cron' mode

set -e
set -u

type htcacheclean > /dev/null 2>&1 || exit 0
[ -e /etc/default/apache-htcacheclean ] || exit 0


# edit /etc/default/apache-htcacheclean to change this
HTCACHECLEAN_MODE=daemon
HTCACHECLEAN_RUN=auto
HTCACHECLEAN_SIZE=300M
HTCACHECLEAN_PATH=/var/cache/apache2/mod_cache_disk
HTCACHECLEAN_OPTIONS=""

. /etc/default/apache-htcacheclean

[ "$HTCACHECLEAN_MODE" = "cron" ] || exit 0

htcacheclean ${HTCACHECLEAN_OPTIONS}    \
        -p${HTCACHECLEAN_PATH}    \
        -l${HTCACHECLEAN_SIZE}




I think there's a trick to batch them if there's very many. But it
sounds like it's constantly doing rotation.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org





--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.
Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
miked at dewhirst
Nov 25, 2022, 4:41 PM
Post #9 of 14 (998 views)
Permalink
On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
> Some script is probably hogging the workers and not letting them close
> so if you must, just restart, but judging by the amount of traffic you
> seem to have at the time I would truncate in logrotate and not restart
> or reload the server at all.

I run a Python script on the production machine to delete the entire
site (including directory structure) then redeploy everything from the
repo whenever anything changes.

On the basis that Apache (mod-wsgi) only reads the files once on reload,
if they also had the dead finger of my script or perhaps the Python
interpreter hanging on, that may well have "hogged" the workers.

That script now restarts Apache instead of reload.

I have now adjusted the logrotate script to avoid reload/restart logging
status as follows:

/var/log/apache2/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    copytruncate
    sharedscripts
    prerotate
    if [ -d /etc/logrotate.d/httpd-prerotate ]; then
        run-parts /etc/logrotate.d/httpd-prerotate
    fi
    endscript
    postrotate
    if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
        invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
    fi
    endscript
}


... and left the mpm-event settings as follows:

<IfModule mpm_event_module>
        ServerLimit                     32                 # Thanks
Eric Covener
        StartServers                    16
        MinSpareThreads                 400
        MaxSpareThreads                 250
        ThreadLimit                     64
        ThreadsPerChild                 50
        AsyncRequestWorkerFactor        2
        MaxRequestWorkers               800
        MaxConnectionsPerChild          0
</IfModule>

I restarted the entire machine

Next logrotate is 00:00 UTC -- I'll report tomorrow.

Cheers

Mike

--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attached Files:

Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
miked at dewhirst
Nov 27, 2022, 5:09 PM
Post #10 of 14 (956 views)
Permalink
On 26/11/2022 11:41 am, Mike Dewhirst wrote:
> On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
>> Some script is probably hogging the workers and not letting them
>> close so if you must, just restart, but judging by the amount of
>> traffic you seem to have at the time I would truncate in logrotate
>> and not restart or reload the server at all.
>
> I run a Python script on the production machine to delete the entire
> site (including directory structure) then redeploy everything from the
> repo whenever anything changes.
>
> On the basis that Apache (mod-wsgi) only reads the files once on
> reload, if they also had the dead finger of my script or perhaps the
> Python interpreter hanging on, that may well have "hogged" the workers.
>
> That script now restarts Apache instead of reload.
>
> I have now adjusted the logrotate script to avoid reload/restart
> logging status as follows:
>
> /var/log/apache2/*.log {
>     daily
>     missingok
>     rotate 14
>     compress
>     delaycompress
>     notifempty
>     copytruncate
>     sharedscripts
>     prerotate
>     if [ -d /etc/logrotate.d/httpd-prerotate ]; then
>         run-parts /etc/logrotate.d/httpd-prerotate
>     fi
>     endscript
>     postrotate
>     if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
>         invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
>     fi
>     endscript
> }
>
>
> ... and left the mpm-event settings as follows:
>
> <IfModule mpm_event_module>
>         ServerLimit                     32                 # Thanks
> Eric Covener
>         StartServers                    16
>         MinSpareThreads                 400
>         MaxSpareThreads                 250
>         ThreadLimit                     64
>         ThreadsPerChild                 50
>         AsyncRequestWorkerFactor        2
>         MaxRequestWorkers               800
>         MaxConnectionsPerChild          0
> </IfModule>
>
> I restarted the entire machine
>
> Next logrotate is 00:00 UTC -- I'll report tomorrow.

Zero bytes in the error log since Saturday when I sent the above email.

That covers the period with copytruncate in the Apache logrotate conf.

I haven't updated the site so no restarts from me.

I'll adjust the deployment script back to reload instead of restart
before my next deployment and report back.

> Cheers
>
> Mike
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.

--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attached Files:

Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
miked at dewhirst
Nov 27, 2022, 8:27 PM
Post #11 of 14 (952 views)
Permalink
On 28/11/2022 12:09 pm, Mike Dewhirst wrote:
> On 26/11/2022 11:41 am, Mike Dewhirst wrote:
>> On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
>>> Some script is probably hogging the workers and not letting them
>>> close so if you must, just restart, but judging by the amount of
>>> traffic you seem to have at the time I would truncate in logrotate
>>> and not restart or reload the server at all.
>>
>> I run a Python script on the production machine to delete the entire
>> site (including directory structure) then redeploy everything from
>> the repo whenever anything changes.
>>
>> On the basis that Apache (mod-wsgi) only reads the files once on
>> reload, if they also had the dead finger of my script or perhaps the
>> Python interpreter hanging on, that may well have "hogged" the workers.
>>
>> That script now restarts Apache instead of reload.
>>
>> I have now adjusted the logrotate script to avoid reload/restart
>> logging status as follows:
>>
>> /var/log/apache2/*.log {
>>     daily
>>     missingok
>>     rotate 14
>>     compress
>>     delaycompress
>>     notifempty
>>     copytruncate
>>     sharedscripts
>>     prerotate
>>     if [ -d /etc/logrotate.d/httpd-prerotate ]; then
>>         run-parts /etc/logrotate.d/httpd-prerotate
>>     fi
>>     endscript
>>     postrotate
>>     if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
>>         invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
>>     fi
>>     endscript
>> }
>>
>>
>> ... and left the mpm-event settings as follows:
>>
>> <IfModule mpm_event_module>
>>         ServerLimit                     32                 # Thanks
>> Eric Covener
>>         StartServers                    16
>>         MinSpareThreads                 400
>>         MaxSpareThreads                 250
>>         ThreadLimit                     64
>>         ThreadsPerChild                 50
>>         AsyncRequestWorkerFactor        2
>>         MaxRequestWorkers               800
>>         MaxConnectionsPerChild          0
>> </IfModule>
>>
>> I restarted the entire machine
>>
>> Next logrotate is 00:00 UTC -- I'll report tomorrow.
>
> Zero bytes in the error log since Saturday when I sent the above email.
>
> That covers the period with copytruncate in the Apache logrotate conf.
>
> I haven't updated the site so no restarts from me.
>
> I'll adjust the deployment script back to reload instead of restart
> before my next deployment and report back.

Errors recommence as follows:

... not an error but the reload after a deployment script run...

[Mon Nov 28 01:17:06.605572 2022] [mpm_event:notice] [pid 920:tid
140638735415168] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 28 01:17:06.699315 2022] [mpm_event:notice] [pid 920:tid
140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Mon Nov 28 01:17:06.699345 2022] [core:notice] [pid 920:tid
140638735415168] AH00094: Command line: '/usr/sbin/apache2'

... lengthy period ensues with no errors followed by another deployment
reload ...

[Mon Nov 28 03:16:44.368563 2022] [mpm_event:notice] [pid 920:tid
140638735415168] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 28 03:16:44.502990 2022] [mpm_event:notice] [pid 920:tid
140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Mon Nov 28 03:16:44.503027 2022] [core:notice] [pid 920:tid
140638735415168] AH00094: Command line: '/usr/sbin/apache2'

... another deployment script reload after a period with no errors ...

[Mon Nov 28 04:08:02.215591 2022] [mpm_event:notice] [pid 920:tid
140638735415168] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Nov 28 04:08:02.319672 2022] [mpm_event:notice] [pid 920:tid
140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Mon Nov 28 04:08:02.319713 2022] [core:notice] [pid 920:tid
140638735415168] AH00094: Command line: '/usr/sbin/apache2'

... but two seconds later ...

[Mon Nov 28 04:08:04.324587 2022] [mpm_event:error] [pid 920:tid
140638735415168] AH03490: scoreboard is full, not at
MaxRequestWorkers.Increase ServerLimit.

... followed by more of the same at approximately 1 second intervals ...

[Mon Nov 28 04:15:30.915730 2022] [mpm_event:error] [pid 920:tid
140638735415168] AH03490: scoreboard is full, not at
MaxRequestWorkers.Increase ServerLimit.

About to revert my deployment script to restart Apache instead of reload ...

Not sure what else I can do?

Cheers

Mike


>
>> Cheers
>>
>> Mike
>>
>> --
>> Signed email is an absolute defence against phishing. This email has
>> been signed with my private key. If you import my public key you can
>> automatically decrypt my signature and be sure it came from me. Just
>> ask and I'll send it to you. Your email software can handle signing.
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.

--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.

Attached Files:

Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
dferradal at apache
Nov 28, 2022, 12:52 PM
Post #12 of 14 (942 views)
Permalink
Hello,

In case something hogs the workers, until you figure out what and solve it:

MaxConnectionsPerChild 0 # inifinite connections per child so apache
does not try to gracefully shut them down.
MaxSpareThreads = MaxRequestWorkers

Never do a graceful restart.

This is what I had to do in production with certain older version of
apache in which mod_weblogic would cause the same behaviour you
describe

Cheers

El lun, 28 nov 2022 a las 5:28, Mike Dewhirst
(<miked@dewhirst.com.au>) escribió:
>
> On 28/11/2022 12:09 pm, Mike Dewhirst wrote:
>
> On 26/11/2022 11:41 am, Mike Dewhirst wrote:
>
> On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
>
> Some script is probably hogging the workers and not letting them close so if you must, just restart, but judging by the amount of traffic you seem to have at the time I would truncate in logrotate and not restart or reload the server at all.
>
>
> I run a Python script on the production machine to delete the entire site (including directory structure) then redeploy everything from the repo whenever anything changes.
>
> On the basis that Apache (mod-wsgi) only reads the files once on reload, if they also had the dead finger of my script or perhaps the Python interpreter hanging on, that may well have "hogged" the workers.
>
> That script now restarts Apache instead of reload.
>
> I have now adjusted the logrotate script to avoid reload/restart logging status as follows:
>
> /var/log/apache2/*.log {
> daily
> missingok
> rotate 14
> compress
> delaycompress
> notifempty
> copytruncate
> sharedscripts
> prerotate
> if [ -d /etc/logrotate.d/httpd-prerotate ]; then
> run-parts /etc/logrotate.d/httpd-prerotate
> fi
> endscript
> postrotate
> if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
> invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
> fi
> endscript
> }
>
>
> ... and left the mpm-event settings as follows:
>
> <IfModule mpm_event_module>
> ServerLimit 32 # Thanks Eric Covener
> StartServers 16
> MinSpareThreads 400
> MaxSpareThreads 250
> ThreadLimit 64
> ThreadsPerChild 50
> AsyncRequestWorkerFactor 2
> MaxRequestWorkers 800
> MaxConnectionsPerChild 0
> </IfModule>
>
> I restarted the entire machine
>
> Next logrotate is 00:00 UTC -- I'll report tomorrow.
>
>
> Zero bytes in the error log since Saturday when I sent the above email.
>
> That covers the period with copytruncate in the Apache logrotate conf.
>
> I haven't updated the site so no restarts from me.
>
> I'll adjust the deployment script back to reload instead of restart before my next deployment and report back.
>
>
> Errors recommence as follows:
>
> ... not an error but the reload after a deployment script run...
>
> [Mon Nov 28 01:17:06.605572 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 01:17:06.699315 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 01:17:06.699345 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... lengthy period ensues with no errors followed by another deployment reload ...
>
> [Mon Nov 28 03:16:44.368563 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 03:16:44.502990 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 03:16:44.503027 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... another deployment script reload after a period with no errors ...
>
> [Mon Nov 28 04:08:02.215591 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 04:08:02.319672 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 04:08:02.319713 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... but two seconds later ...
>
> [Mon Nov 28 04:08:04.324587 2022] [mpm_event:error] [pid 920:tid 140638735415168] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> ... followed by more of the same at approximately 1 second intervals ...
>
> [Mon Nov 28 04:15:30.915730 2022] [mpm_event:error] [pid 920:tid 140638735415168] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> About to revert my deployment script to restart Apache instead of reload ...
>
> Not sure what else I can do?
>
> Cheers
>
> Mike
>
>
>
> Cheers
>
> Mike
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.



--
Daniel Ferradal
HTTPD Project
#httpd help at Libera.Chat

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
julien.tamil at gmail
Nov 28, 2022, 11:50 PM
Post #13 of 14 (926 views)
Permalink
Ok, thanks for this advice Daniel.


Le lun. 28 nov. 2022 à 21:53, Daniel Ferradal <dferradal@apache.org> a
écrit :

> Hello,
>
> In case something hogs the workers, until you figure out what and solve it:
>
> MaxConnectionsPerChild 0 # inifinite connections per child so apache
> does not try to gracefully shut them down.
> MaxSpareThreads = MaxRequestWorkers
>
> Never do a graceful restart.
>
> This is what I had to do in production with certain older version of
> apache in which mod_weblogic would cause the same behaviour you
> describe
>
> Cheers
>
> El lun, 28 nov 2022 a las 5:28, Mike Dewhirst
> (<miked@dewhirst.com.au>) escribió:
> >
> > On 28/11/2022 12:09 pm, Mike Dewhirst wrote:
> >
> > On 26/11/2022 11:41 am, Mike Dewhirst wrote:
> >
> > On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
> >
> > Some script is probably hogging the workers and not letting them close
> so if you must, just restart, but judging by the amount of traffic you seem
> to have at the time I would truncate in logrotate and not restart or reload
> the server at all.
> >
> >
> > I run a Python script on the production machine to delete the entire
> site (including directory structure) then redeploy everything from the repo
> whenever anything changes.
> >
> > On the basis that Apache (mod-wsgi) only reads the files once on reload,
> if they also had the dead finger of my script or perhaps the Python
> interpreter hanging on, that may well have "hogged" the workers.
> >
> > That script now restarts Apache instead of reload.
> >
> > I have now adjusted the logrotate script to avoid reload/restart logging
> status as follows:
> >
> > /var/log/apache2/*.log {
> > daily
> > missingok
> > rotate 14
> > compress
> > delaycompress
> > notifempty
> > copytruncate
> > sharedscripts
> > prerotate
> > if [ -d /etc/logrotate.d/httpd-prerotate ]; then
> > run-parts /etc/logrotate.d/httpd-prerotate
> > fi
> > endscript
> > postrotate
> > if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
> > invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
> > fi
> > endscript
> > }
> >
> >
> > ... and left the mpm-event settings as follows:
> >
> > <IfModule mpm_event_module>
> > ServerLimit 32 # Thanks Eric
> Covener
> > StartServers 16
> > MinSpareThreads 400
> > MaxSpareThreads 250
> > ThreadLimit 64
> > ThreadsPerChild 50
> > AsyncRequestWorkerFactor 2
> > MaxRequestWorkers 800
> > MaxConnectionsPerChild 0
> > </IfModule>
> >
> > I restarted the entire machine
> >
> > Next logrotate is 00:00 UTC -- I'll report tomorrow.
> >
> >
> > Zero bytes in the error log since Saturday when I sent the above email.
> >
> > That covers the period with copytruncate in the Apache logrotate conf.
> >
> > I haven't updated the site so no restarts from me.
> >
> > I'll adjust the deployment script back to reload instead of restart
> before my next deployment and report back.
> >
> >
> > Errors recommence as follows:
> >
> > ... not an error but the reload after a deployment script run...
> >
> > [Mon Nov 28 01:17:06.605572 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> > [Mon Nov 28 01:17:06.699315 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
> mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> > [Mon Nov 28 01:17:06.699345 2022] [core:notice] [pid 920:tid
> 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
> >
> > ... lengthy period ensues with no errors followed by another deployment
> reload ...
> >
> > [Mon Nov 28 03:16:44.368563 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> > [Mon Nov 28 03:16:44.502990 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
> mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> > [Mon Nov 28 03:16:44.503027 2022] [core:notice] [pid 920:tid
> 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
> >
> > ... another deployment script reload after a period with no errors ...
> >
> > [Mon Nov 28 04:08:02.215591 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> > [Mon Nov 28 04:08:02.319672 2022] [mpm_event:notice] [pid 920:tid
> 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2
> mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> > [Mon Nov 28 04:08:02.319713 2022] [core:notice] [pid 920:tid
> 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
> >
> > ... but two seconds later ...
> >
> > [Mon Nov 28 04:08:04.324587 2022] [mpm_event:error] [pid 920:tid
> 140638735415168] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
> >
> > ... followed by more of the same at approximately 1 second intervals ...
> >
> > [Mon Nov 28 04:15:30.915730 2022] [mpm_event:error] [pid 920:tid
> 140638735415168] AH03490: scoreboard is full, not at
> MaxRequestWorkers.Increase ServerLimit.
> >
> > About to revert my deployment script to restart Apache instead of reload
> ...
> >
> > Not sure what else I can do?
> >
> > Cheers
> >
> > Mike
> >
> >
> >
> > Cheers
> >
> > Mike
> >
> > --
> > Signed email is an absolute defence against phishing. This email has
> > been signed with my private key. If you import my public key you can
> > automatically decrypt my signature and be sure it came from me. Just
> > ask and I'll send it to you. Your email software can handle signing.
> >
> >
> > --
> > Signed email is an absolute defence against phishing. This email has
> > been signed with my private key. If you import my public key you can
> > automatically decrypt my signature and be sure it came from me. Just
> > ask and I'll send it to you. Your email software can handle signing.
> >
> >
> > --
> > Signed email is an absolute defence against phishing. This email has
> > been signed with my private key. If you import my public key you can
> > automatically decrypt my signature and be sure it came from me. Just
> > ask and I'll send it to you. Your email software can handle signing.
>
>
>
> --
> Daniel Ferradal
> HTTPD Project
> #httpd help at Libera.Chat
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
RE: Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit [ In reply to ]
sierra.padilla at excitedly
Dec 10, 2022, 3:54 AM
Post #14 of 14 (789 views)
Permalink
Hi,
Just checking, are you interested acquiring the list of attendees? Please respond my email. I'm waiting for your response.
We are discussing about Prices And, provide the discount cost.
Thank you.



From: Ju lien <julien.tamil@gmail.com>
Sent: Tuesday, November 29, 2022 1:21 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Error AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit

Ok, thanks for this advice Daniel.


Le lun. 28 nov. 2022 à 21:53, Daniel Ferradal <dferradal@apache.org<mailto:dferradal@apache.org>> a écrit :
Hello,

In case something hogs the workers, until you figure out what and solve it:

MaxConnectionsPerChild 0 # inifinite connections per child so apache
does not try to gracefully shut them down.
MaxSpareThreads = MaxRequestWorkers

Never do a graceful restart.

This is what I had to do in production with certain older version of
apache in which mod_weblogic would cause the same behaviour you
describe

Cheers

El lun, 28 nov 2022 a las 5:28, Mike Dewhirst
(<miked@dewhirst.com.au<mailto:miked@dewhirst.com.au>>) escribió:
>
> On 28/11/2022 12:09 pm, Mike Dewhirst wrote:
>
> On 26/11/2022 11:41 am, Mike Dewhirst wrote:
>
> On 24/11/2022 6:48 pm, Daniel Ferradal wrote:
>
> Some script is probably hogging the workers and not letting them close so if you must, just restart, but judging by the amount of traffic you seem to have at the time I would truncate in logrotate and not restart or reload the server at all.
>
>
> I run a Python script on the production machine to delete the entire site (including directory structure) then redeploy everything from the repo whenever anything changes.
>
> On the basis that Apache (mod-wsgi) only reads the files once on reload, if they also had the dead finger of my script or perhaps the Python interpreter hanging on, that may well have "hogged" the workers.
>
> That script now restarts Apache instead of reload.
>
> I have now adjusted the logrotate script to avoid reload/restart logging status as follows:
>
> /var/log/apache2/*.log {
> daily
> missingok
> rotate 14
> compress
> delaycompress
> notifempty
> copytruncate
> sharedscripts
> prerotate
> if [ -d /etc/logrotate.d/httpd-prerotate ]; then
> run-parts /etc/logrotate.d/httpd-prerotate
> fi
> endscript
> postrotate
> if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
> invoke-rc.d apache2 status 2>&1 | logger -t apache2.logrotate
> fi
> endscript
> }
>
>
> ... and left the mpm-event settings as follows:
>
> <IfModule mpm_event_module>
> ServerLimit 32 # Thanks Eric Covener
> StartServers 16
> MinSpareThreads 400
> MaxSpareThreads 250
> ThreadLimit 64
> ThreadsPerChild 50
> AsyncRequestWorkerFactor 2
> MaxRequestWorkers 800
> MaxConnectionsPerChild 0
> </IfModule>
>
> I restarted the entire machine
>
> Next logrotate is 00:00 UTC -- I'll report tomorrow.
>
>
> Zero bytes in the error log since Saturday when I sent the above email.
>
> That covers the period with copytruncate in the Apache logrotate conf.
>
> I haven't updated the site so no restarts from me.
>
> I'll adjust the deployment script back to reload instead of restart before my next deployment and report back.
>
>
> Errors recommence as follows:
>
> ... not an error but the reload after a deployment script run...
>
> [Mon Nov 28 01:17:06.605572 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 01:17:06.699315 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 01:17:06.699345 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... lengthy period ensues with no errors followed by another deployment reload ...
>
> [Mon Nov 28 03:16:44.368563 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 03:16:44.502990 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 03:16:44.503027 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... another deployment script reload after a period with no errors ...
>
> [Mon Nov 28 04:08:02.215591 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00493: SIGUSR1 received. Doing graceful restart
> [Mon Nov 28 04:08:02.319672 2022] [mpm_event:notice] [pid 920:tid 140638735415168] AH00489: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
> [Mon Nov 28 04:08:02.319713 2022] [core:notice] [pid 920:tid 140638735415168] AH00094: Command line: '/usr/sbin/apache2'
>
> ... but two seconds later ...
>
> [Mon Nov 28 04:08:04.324587 2022] [mpm_event:error] [pid 920:tid 140638735415168] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> ... followed by more of the same at approximately 1 second intervals ...
>
> [Mon Nov 28 04:15:30.915730 2022] [mpm_event:error] [pid 920:tid 140638735415168] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> About to revert my deployment script to restart Apache instead of reload ...
>
> Not sure what else I can do?
>
> Cheers
>
> Mike
>
>
>
> Cheers
>
> Mike
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.
>
>
> --
> Signed email is an absolute defence against phishing. This email has
> been signed with my private key. If you import my public key you can
> automatically decrypt my signature and be sure it came from me. Just
> ask and I'll send it to you. Your email software can handle signing.



--
Daniel Ferradal
HTTPD Project
#httpd help at Libera.Chat

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org<mailto:users-unsubscribe@httpd.apache.org>
For additional commands, e-mail: users-help@httpd.apache.org<mailto:users-help@httpd.apache.org>

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal