Mailing List Archive

Runaway CPU and memory on high load
Would it be possible to combile backhand into apache on that machine
and attach GDB to one of the run-away processes. I've heard of that
problem before. It appears to be some sort of infinite recursion (as I
don't see how infinite loops would occur in the code). If you can gdb
it, break it and give me a full backtrace I will try to get to the
bottom of the problem.

On Tuesday, May 27, 2003, at 17:37 US/Eastern, Brian McCashin wrote:

> I've been running backhand 1.2.2 and apache 1.3.27/php 4.3.0
> RH7.1(2.4.2-2) tenuously
> for 2 weeks now. (It was a backup plan for a failed Aeropoint.)
> My moderator handles ~600 hits per second at the daily peak
> to four backend servers all (except 1) Dual 800 Xeons.
>
> The problem is that after a short time 1 or 2 apache processes start
> growing
> and sucking up CPU. If got a cron'ed job to kill processes if they
> exceed
> 15Meg in VM but without this the moderator would soon run out of
> memory.
> Config follows.
> Also I don't use byBusychildren because one of the backend never show
> servers
> in use and gets all new traffic.
>
> I'm renicing the moderator and the moderator has keepalives off
> whereas the rest are on.
> Any ideas??
>
> <IfModule mod_backhand.c>
> UnixSocketDir /usr/local/apache/backhand
> MulticastStats 10.0.0.200 10.0.0.255:4445
> AcceptStats 10.0.0.0/24
> BackhandModeratorPIDFile /usr/local/apache/backhand/moderator.pid
> BackhandConnectionPools off
> <Location "/backhand">
> SetHandler backhand-handler
> </Location>
> </IfModule>
>
> <Directory "/usr/local/apache/htdocs">
> Backhand removeSelf
> Backhand byAge
> Backhand bySession
> </Directory>
>
>
>
>
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>
--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
Runaway CPU and memory on high load [ In reply to ]
I've never gdb'ed an apache module. Do I have to only compile symbols
into the backhand module or apache as well
to get a usable backtrace.

On Tuesday, June 3, 2003, at 08:36 AM, Theo Schlossnagle wrote:

> Would it be possible to combile backhand into apache on that machine
> and attach GDB to one of the run-away processes. I've heard of that
> problem before. It appears to be some sort of infinite recursion (as I
> don't see how infinite loops would occur in the code). If you can gdb
> it, break it and give me a full backtrace I will try to get to the
> bottom of the problem.
>
> On Tuesday, May 27, 2003, at 17:37 US/Eastern, Brian McCashin wrote:
>
>> I've been running backhand 1.2.2 and apache 1.3.27/php 4.3.0
>> RH7.1(2.4.2-2) tenuously
>> for 2 weeks now. (It was a backup plan for a failed Aeropoint.)
>> My moderator handles ~600 hits per second at the daily peak
>> to four backend servers all (except 1) Dual 800 Xeons.
>>
>> The problem is that after a short time 1 or 2 apache processes start
>> growing
>> and sucking up CPU. If got a cron'ed job to kill processes if they
>> exceed
>> 15Meg in VM but without this the moderator would soon run out of
>> memory.
>> Config follows.
>> Also I don't use byBusychildren because one of the backend never show
>> servers
>> in use and gets all new traffic.
>>
>> I'm renicing the moderator and the moderator has keepalives off
>> whereas the rest are on.
>> Any ideas??
>>
>> <IfModule mod_backhand.c>
>> UnixSocketDir /usr/local/apache/backhand
>> MulticastStats 10.0.0.200 10.0.0.255:4445
>> AcceptStats 10.0.0.0/24
>> BackhandModeratorPIDFile /usr/local/apache/backhand/moderator.pid
>> BackhandConnectionPools off
>> <Location "/backhand">
>> SetHandler backhand-handler
>> </Location>
>> </IfModule>
>>
>> <Directory "/usr/local/apache/htdocs">
>> Backhand removeSelf
>> Backhand byAge
>> Backhand bySession
>> </Directory>
>>
>>
>>
>>
>>
>> _______________________________________________
>> backhand-users mailing list
>> backhand-users@lists.backhand.org
>> http://lists.backhand.org/mailman/listinfo/backhand-users
>>
> -- Theo Schlossnagle
> Principal Consultant
> OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
> Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
> 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
> 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>
Runaway CPU and memory on high load [ In reply to ]
To make your like as easy as possible, I would do this:
fresh untar Apache,
"apply" mod_backhand to that source tree using ./precompile.

Add -g to the CFLAGS and ./configure apache adding mode_backhand
_statically_.
make install
cd src

And: gdb httpd <pid>

You should have a full symbol set at that point. It would be good to
have full symbols in apache as well so that we can account for a
ping-pong infinite recursion between mod_backhand and an Apache
internal call -- should there be one.

On Thursday, Jun 5, 2003, at 19:20 US/Eastern, Brian McCashin wrote:
> I've never gdb'ed an apache module. Do I have to only compile symbols
> into the backhand module or apache as well
> to get a usable backtrace.
>
> On Tuesday, June 3, 2003, at 08:36 AM, Theo Schlossnagle wrote:
>
>> Would it be possible to combile backhand into apache on that machine
>> and attach GDB to one of the run-away processes. I've heard of that
>> problem before. It appears to be some sort of infinite recursion (as
>> I don't see how infinite loops would occur in the code). If you can
>> gdb it, break it and give me a full backtrace I will try to get to
>> the bottom of the problem.
>>
>> On Tuesday, May 27, 2003, at 17:37 US/Eastern, Brian McCashin wrote:
>>
>>> I've been running backhand 1.2.2 and apache 1.3.27/php 4.3.0
>>> RH7.1(2.4.2-2) tenuously
>>> for 2 weeks now. (It was a backup plan for a failed Aeropoint.)
>>> My moderator handles ~600 hits per second at the daily peak
>>> to four backend servers all (except 1) Dual 800 Xeons.
>>>
>>> The problem is that after a short time 1 or 2 apache processes start
>>> growing
>>> and sucking up CPU. If got a cron'ed job to kill processes if they
>>> exceed
>>> 15Meg in VM but without this the moderator would soon run out of
>>> memory.
>>> Config follows.
>>> Also I don't use byBusychildren because one of the backend never
>>> show servers
>>> in use and gets all new traffic.
>>>
>>> I'm renicing the moderator and the moderator has keepalives off
>>> whereas the rest are on.
>>> Any ideas??
>>>
>>> <IfModule mod_backhand.c>
>>> UnixSocketDir /usr/local/apache/backhand
>>> MulticastStats 10.0.0.200 10.0.0.255:4445
>>> AcceptStats 10.0.0.0/24
>>> BackhandModeratorPIDFile /usr/local/apache/backhand/moderator.pid
>>> BackhandConnectionPools off
>>> <Location "/backhand">
>>> SetHandler backhand-handler
>>> </Location>
>>> </IfModule>
>>>
>>> <Directory "/usr/local/apache/htdocs">
>>> Backhand removeSelf
>>> Backhand byAge
>>> Backhand bySession
>>> </Directory>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> backhand-users mailing list
>>> backhand-users@lists.backhand.org
>>> http://lists.backhand.org/mailman/listinfo/backhand-users
>>>
>> -- Theo Schlossnagle
>> Principal Consultant
>> OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
>> Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
>> 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>> 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>>
>
>
--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7