Mailing List Archive: Loosing umbilical to operator

Loosing umbilical to operator

Oct 6, 2002, 11:01 AM

Post #1 of 21 (11758 views)

Hello to all on the list.

For the last two months I have had a problem with backhand running
reliably. Though backhand starts up with no problem and runs for hours,
after anywhere from 4 hours to two days, backhand starts serving and
starts placing a gratuitious number of the following error messages into
my apache error log file:

[Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
failed to establish umbilical to moderator!

This is a "dumb front end" tier 1 box only has a single html file
basically stating that the site is down and to come back later. The rest
of the website (www.linuxguru.net) is actually on a backend box (normally
two, but one threw a hard drive)

I am using the following versions of software:
---cust here
name developer debian
version version
backhand 1.2.1 1.2.1-1.2
apache 1.3.26 1.3.26-1.1
Linux 2.4.18 n/a
---cust here

The backhand pertinant part of my httpd.conf looks thusly:
---cut here
LoadModule backhand_module /usr/lib/apache/1.3/mod_backhand.so
AddModule mod_backhand.c
UnixSocketDir /var/run/libapache-mod-backhand
MulticastStats 192.168.2.1:4445
AcceptStats 192.168.2.0/24

<Location "/backhand/">
SetHandler backhand-handler
</Location>
---cut here

When things are normally working, a ps aux shows apache processes along
these lines:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 8129 0.0 1.2 3360 1628 ? S 13:19 0:00 /usr/sbin/apache

I have noticed that on occasion these processes jump up in their RSS size
all the way up to 32 megabytes occasionally. I think that this is somehow
related but I am not sure. (To be honest, I suspected that its a memory
leak caused when the umbilical is lost.)

Oh, one other thing. I am unable to reproduce this at will, though it
happens frequently enough (anywhere from semidaily to bidaily) that I can
look into things while it's not working. When I stress test the site, it works
fine, so the umbilical loss seems to not be related to load.

General stats for the site:
---cut here
- Average page build on the backend machine is .60 seconds
- Usual page size (including images) around 80k
- Frontend server is a Athlon 500 w/ 128 megs of ram
- Frontend server has 128 megs of ram.
- Site gets approximately 1500 page views a day
---cut here

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

george at omniti

Oct 6, 2002, 11:14 AM

Post #2 of 21 (11643 views)

Permalink

Is the backhand bparent file perhaps being deleted by a cron'd 'cleanup'
script.

George

On Sunday, October 6, 2002, at 02:01 PM, newsgate@linuxguru.net wrote:

>
> Hello to all on the list.
>
> For the last two months I have had a problem with backhand running
> reliably. Though backhand starts up with no problem and runs for hours,
> after anywhere from 4 hours to two days, backhand starts serving and
> starts placing a gratuitious number of the following error messages into
> my apache error log file:
>
> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
> failed to establish umbilical to moderator!
>
> This is a "dumb front end" tier 1 box only has a single html file
> basically stating that the site is down and to come back later. The rest
> of the website (www.linuxguru.net) is actually on a backend box
> (normally
> two, but one threw a hard drive)
>
> I am using the following versions of software:
> ---cust here
> name developer debian
> version version
> backhand 1.2.1 1.2.1-1.2
> apache 1.3.26 1.3.26-1.1
> Linux 2.4.18 n/a
> ---cust here
>
>
> The backhand pertinant part of my httpd.conf looks thusly:
> ---cut here
> LoadModule backhand_module /usr/lib/apache/1.3/mod_backhand.so
> AddModule mod_backhand.c
> UnixSocketDir /var/run/libapache-mod-backhand
> MulticastStats 192.168.2.1:4445
> AcceptStats 192.168.2.0/24
>
> <Location "/backhand/">
> SetHandler backhand-handler
> </Location>
> ---cut here
>
>
> When things are normally working, a ps aux shows apache processes along
> these lines:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> www-data 8129 0.0 1.2 3360 1628 ? S 13:19 0:00
> /usr/sbin/apache
>
> I have noticed that on occasion these processes jump up in their RSS
> size
> all the way up to 32 megabytes occasionally. I think that this is
> somehow
> related but I am not sure. (To be honest, I suspected that its a memory
> leak caused when the umbilical is lost.)
>
> Oh, one other thing. I am unable to reproduce this at will, though it
> happens frequently enough (anywhere from semidaily to bidaily) that I
> can
> look into things while it's not working. When I stress test the site,
> it works
> fine, so the umbilical loss seems to not be related to load.
>
> General stats for the site:
> ---cut here
> - Average page build on the backend machine is .60 seconds
> - Usual page size (including images) around 80k
> - Frontend server is a Athlon 500 w/ 128 megs of ram
> - Frontend server has 128 megs of ram.
> - Site gets approximately 1500 page views a day
> ---cut here
>
>
> --
> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
> James Blackwell -- Director http://www.linuxguru.net
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>
// George Schlossnagle
// Principal Consultant
// OmniTI, Inc http://www.omniti.com
// (c) 240.460.5234 (e) george@omniti.com
// 1024D/1100A5A0 1370 F70A 9365 96C9 2F5E 56C2 B2B9 262F 1100 A5A0

Loosing umbilical to operator [ In reply to ]

rama at pitties

Oct 6, 2002, 11:23 AM

Post #3 of 21 (11656 views)

Permalink

Are you reniceing your moderator? I ran into this under
high load till I did something like the following:

renice -20 `cat /usr/local/apache/backhand/moderator.pid`
(note: you would also need a line in your conf like the
following:
BackhandModeratorPIDFile /usr/local/apache/backhand/moderator.pid)

Hope this helps,
-Rama

newsgate@linuxguru.net wrote:

>Hello to all on the list.
>
>For the last two months I have had a problem with backhand running
>reliably. Though backhand starts up with no problem and runs for hours,
>after anywhere from 4 hours to two days, backhand starts serving and
>starts placing a gratuitious number of the following error messages into
>my apache error log file:
>
>[Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>failed to establish umbilical to moderator!
>
>This is a "dumb front end" tier 1 box only has a single html file
>basically stating that the site is down and to come back later. The rest
>of the website (www.linuxguru.net) is actually on a backend box (normally
>two, but one threw a hard drive)
>
>I am using the following versions of software:
>---cust here
>name developer debian
> version version
>backhand 1.2.1 1.2.1-1.2
>apache 1.3.26 1.3.26-1.1
>Linux 2.4.18 n/a
>---cust here
>
>
>The backhand pertinant part of my httpd.conf looks thusly:
>---cut here
>LoadModule backhand_module /usr/lib/apache/1.3/mod_backhand.so
>AddModule mod_backhand.c
>UnixSocketDir /var/run/libapache-mod-backhand
>MulticastStats 192.168.2.1:4445
>AcceptStats 192.168.2.0/24
>
><Location "/backhand/">
> SetHandler backhand-handler
></Location>
>---cut here
>
>
>When things are normally working, a ps aux shows apache processes along
>these lines:
>
>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>www-data 8129 0.0 1.2 3360 1628 ? S 13:19 0:00 /usr/sbin/apache
>
>I have noticed that on occasion these processes jump up in their RSS size
>all the way up to 32 megabytes occasionally. I think that this is somehow
>related but I am not sure. (To be honest, I suspected that its a memory
>leak caused when the umbilical is lost.)
>
>Oh, one other thing. I am unable to reproduce this at will, though it
>happens frequently enough (anywhere from semidaily to bidaily) that I can
>look into things while it's not working. When I stress test the site, it works
>fine, so the umbilical loss seems to not be related to load.
>
>General stats for the site:
>---cut here
>- Average page build on the backend machine is .60 seconds
>- Usual page size (including images) around 80k
>- Frontend server is a Athlon 500 w/ 128 megs of ram
>- Frontend server has 128 megs of ram.
>- Site gets approximately 1500 page views a day
>---cut here
>
>
>
>

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 6, 2002, 6:07 PM

Post #4 of 21 (11652 views)

Permalink

Hello to all on the list.

For the last two months I have had a problem with backhand running
reliably. Though backhand starts up with no problem and runs for hours,
after anywhere from 4 hours to two days, backhand starts serving and
starts placing a gratuitious number of the following error messages into
my apache error log file:

[Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
failed to establish umbilical to moderator!

This is a "dumb front end" tier 1 box only has a single html file
basically stating that the site is down and to come back later. The rest
of the website (www.linuxguru.net) is actually on a backend box (normally
two, but one threw a hard drive)

I am using the following versions of software:
---cust here
name developer debian
version version
backhand 1.2.1 1.2.1-1.2
apache 1.3.26 1.3.26-1.1
Linux 2.4.18 n/a
---cust here

The backhand pertinant part of my httpd.conf looks thusly:
---cut here
LoadModule backhand_module /usr/lib/apache/1.3/mod_backhand.so
AddModule mod_backhand.c
UnixSocketDir /var/run/libapache-mod-backhand
MulticastStats 192.168.2.1:4445
AcceptStats 192.168.2.0/24

<Location "/backhand/">
SetHandler backhand-handler
</Location>
---cut here

When things are normally working, a ps aux shows apache processes along
these lines:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 8129 0.0 1.2 3360 1628 ? S 13:19 0:00 /usr/sbin/apache

When backhand stops working, the rss _shrinks_ -- usually down to about
350 or so. I think that this is somehow
related but I am not sure.

Oh, one other thing. I am unable to reproduce this at will, though it
happens frequently enough (anywhere from semidaily to bidaily) that I can
look into things while it's not working. When I stress test the site, it works
fine, so the umbilical loss seems to not be related to load.

General stats for the site:
---cut here
- Average page build on the backend machine is .60 seconds
- Usual page size (including images) around 80k
- Frontend server is a Athlon 500 w/ 128 megs of ram
- Frontend server has 128 megs of ram.
- Site gets approximately 1500 page views a day
---cut here

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 6, 2002, 6:50 PM

Post #5 of 21 (11676 views)

Permalink

In lists.backhand.users, you wrote:
> Is the backhand bparent file perhaps being deleted by a cron'd 'cleanup'
> script.

No. bparent remains.

>
> George
>
> On Sunday, October 6, 2002, at 02:01 PM, newsgate@linuxguru.net wrote:
>
>>
>> Hello to all on the list.
>>
>> For the last two months I have had a problem with backhand running
>> reliably. Though backhand starts up with no problem and runs for hours,
>> after anywhere from 4 hours to two days, backhand starts serving and
>> starts placing a gratuitious number of the following error messages into
>> my apache error log file:
>>
>> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>> failed to establish umbilical to moderator!
>>
>> This is a "dumb front end" tier 1 box only has a single html file
>> basically stating that the site is down and to come back later. The rest
>> of the website (www.linuxguru.net) is actually on a backend box
>> (normally
>> two, but one threw a hard drive)
>>
>> I am using the following versions of software:
>> ---cust here
>> name developer debian
>> version version
>> backhand 1.2.1 1.2.1-1.2
>> apache 1.3.26 1.3.26-1.1
>> Linux 2.4.18 n/a
>> ---cust here
>>
>>
>> The backhand pertinant part of my httpd.conf looks thusly:
>> ---cut here
>> LoadModule backhand_module /usr/lib/apache/1.3/mod_backhand.so
>> AddModule mod_backhand.c
>> UnixSocketDir /var/run/libapache-mod-backhand
>> MulticastStats 192.168.2.1:4445
>> AcceptStats 192.168.2.0/24
>>
>> <Location "/backhand/">
>> SetHandler backhand-handler
>> </Location>
>> ---cut here
>>
>>
>> When things are normally working, a ps aux shows apache processes along
>> these lines:
>>
>> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>> www-data 8129 0.0 1.2 3360 1628 ? S 13:19 0:00
>> /usr/sbin/apache
>>
>> I have noticed that on occasion these processes jump up in their RSS
>> size
>> all the way up to 32 megabytes occasionally. I think that this is
>> somehow
>> related but I am not sure. (To be honest, I suspected that its a memory
>> leak caused when the umbilical is lost.)
>>
>> Oh, one other thing. I am unable to reproduce this at will, though it
>> happens frequently enough (anywhere from semidaily to bidaily) that I
>> can
>> look into things while it's not working. When I stress test the site,
>> it works
>> fine, so the umbilical loss seems to not be related to load.
>>
>> General stats for the site:
>> ---cut here
>> - Average page build on the backend machine is .60 seconds
>> - Usual page size (including images) around 80k
>> - Frontend server is a Athlon 500 w/ 128 megs of ram
>> - Frontend server has 128 megs of ram.
>> - Site gets approximately 1500 page views a day
>> ---cut here
>>
>>
>> --
>> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
>> James Blackwell -- Director http://www.linuxguru.net
>>
>> _______________________________________________
>> backhand-users mailing list
>> backhand-users@lists.backhand.org
>> http://lists.backhand.org/mailman/listinfo/backhand-users
>>
> // George Schlossnagle
> // Principal Consultant
> // OmniTI, Inc http://www.omniti.com
> // (c) 240.460.5234 (e) george@omniti.com
> // 1024D/1100A5A0 1370 F70A 9365 96C9 2F5E 56C2 B2B9 262F 1100 A5A0
>
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 6, 2002, 6:50 PM

Post #6 of 21 (11676 views)

Permalink

In lists.backhand.users, you wrote:
> Are you reniceing your moderator? I ran into this under
> high load till I did something like the following:
>
> renice -20 `cat /usr/local/apache/backhand/moderator.pid`
> (note: you would also need a line in your conf like the
> following:
> BackhandModeratorPIDFile /usr/local/apache/backhand/moderator.pid)

I'll give it a try, but system load on this machine is nearly always 0.

>
> Hope this helps,
> -Rama
>
> newsgate@linuxguru.net wrote:
>
>>Hello to all on the list.
>>
>>For the last two months I have had a problem with backhand running
>>reliably. Though backhand starts up with no problem and runs for hours,
>>after anywhere from 4 hours to two days, backhand starts serving and
>>starts placing a gratuitious number of the following error messages into
>>my apache error log file:
>>
>>[Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>>failed to establish umbilical to moderator!
>>
>>This is a "dumb front end" tier 1 box only has a single html file
>>basically stating that the site is down and to come back later. The rest
>>of the website (www.linuxguru.net) is actually on a backend box (normally
>>two, but one threw a hard drive)
>>
>>I am using the following versions of software:
>>---cust here
>>name developer debian
>> version version
>>backhand 1.2.1 1.2.1-1.2
>>apache 1.3.26 1.3.26-1.1
>>Linux 2.4.18 n/a
>>---cust here
>>
>>
>>The backhand pertinant part of my httpd.conf looks thusly:
>>---cut here
>>LoadModule backhand_module /usr/lib/apache/1.3/mod_backhand.so
>>AddModule mod_backhand.c
>>UnixSocketDir /var/run/libapache-mod-backhand
>>MulticastStats 192.168.2.1:4445
>>AcceptStats 192.168.2.0/24
>>
>><Location "/backhand/">
>> SetHandler backhand-handler
>></Location>
>>---cut here
>>
>>
>>When things are normally working, a ps aux shows apache processes along
>>these lines:
>>
>>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>>www-data 8129 0.0 1.2 3360 1628 ? S 13:19 0:00 /usr/sbin/apache
>>
>>I have noticed that on occasion these processes jump up in their RSS size
>>all the way up to 32 megabytes occasionally. I think that this is somehow
>>related but I am not sure. (To be honest, I suspected that its a memory
>>leak caused when the umbilical is lost.)
>>
>>Oh, one other thing. I am unable to reproduce this at will, though it
>>happens frequently enough (anywhere from semidaily to bidaily) that I can
>>look into things while it's not working. When I stress test the site, it works
>>fine, so the umbilical loss seems to not be related to load.
>>
>>General stats for the site:
>>---cut here
>>- Average page build on the backend machine is .60 seconds
>>- Usual page size (including images) around 80k
>>- Frontend server is a Athlon 500 w/ 128 megs of ram
>>- Frontend server has 128 megs of ram.
>>- Site gets approximately 1500 page views a day
>>---cut here
>>
>>
>>
>>
>
>
>
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

jesus at omniti

Oct 6, 2002, 8:04 PM

Post #7 of 21 (11662 views)

Permalink

On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
> For the last two months I have had a problem with backhand running
> reliably. Though backhand starts up with no problem and runs for hours,
> after anywhere from 4 hours to two days, backhand starts serving and
> starts placing a gratuitious number of the following error messages into
> my apache error log file:
>
> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
> failed to establish umbilical to moderator!

Sounds to me like the moderator is crashing. Look in your logs fora
httpd segfault or something to that effect immediately before the errors
start up.

I have had reports of this before. Check the /backhand/ page. Is
everything there marked red? This would be a good sign that the
moderator isn't running. If it crashes, this is obviously a bug. I
have been unable to replicate this on any of my installations (running
for months on end with no problems).

--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

---------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
---------------------------------------------------------

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 7, 2002, 7:27 AM

Post #8 of 21 (11658 views)

Permalink

In lists.backhand.users, you wrote:
> In lists.backhand.users, you wrote:
>> Are you reniceing your moderator? I ran into this under
>> high load till I did something like the following:
>>
>> renice -20 `cat /usr/local/apache/backhand/moderator.pid`
>> (note: you would also need a line in your conf like the
>> following:
>> BackhandModeratorPIDFile /usr/local/apache/backhand/moderator.pid)
>
> I'll give it a try, but system load on this machine is nearly always 0.
>>
>> Hope this helps,

No such luck. Any other ideas?

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

leus at epublish

Oct 7, 2002, 8:52 AM

Post #9 of 21 (11633 views)

Permalink

James Blackwell wrote:
> No such luck. Any other ideas?

I have similar problems, and I think we found a pattern in our machines.

We receive about 13 million hits a day (small 1x1 images, we do web
statistics). During the day all our machines work near its max capacity
(num apache childs ~ max servers). But around 7 PM our traffic suddenly
decreases, and I start seeing a lot of these messages in our log file
-umbilica, etc. Recently, I had a misconfiguration (files-max was 8192)
and every evening my front end machines went south. The pattern is the
fact that when the backend machines started killing childs due to the
load decreasing, the front end machines went south.

Of course, I did increase the files-max number and everything is ok now.
I still see the ocassional error message in my log file, but that's all.

Regards,
--
Leonardo Herrera L.
mailto:leus@epublish.cl

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 7, 2002, 1:28 PM

Post #10 of 21 (11658 views)

Permalink

In lists.backhand.users, you wrote:
>
> On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
>> For the last two months I have had a problem with backhand running
>> reliably. Though backhand starts up with no problem and runs for hours,
>> after anywhere from 4 hours to two days, backhand starts serving and
>> starts placing a gratuitious number of the following error messages into
>> my apache error log file:
>>
>> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>> failed to establish umbilical to moderator!
>
> Sounds to me like the moderator is crashing. Look in your logs fora
> httpd segfault or something to that effect immediately before the errors
> start up.

Let me check the logs... Yes. It appears to me that the moderator is
crashing.

recvmsg errorconnection closed by server
[Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
Child 13353 failed to establish umbilical to moderator!

recvmsg errorconnection closed by server
[Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe (13)
[Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
failed to establish umbilical to moderato

I've noticed two things. After the moderator has clearly crashed, the
apache processes are using very, very little memory. an RSS around 332.

However, right at about the time that the moderator crashes or is about
to crash, apache processes on the front end get huge... RSS shows as
ranging anywhere from 20 - 40 megabytes.

Oh, another unusual thing. because of some timing I do, my average page
guild time is .56 seconds, and when backhand isn't being used displays
just about as quickly. But when backhand is involved, though page build
is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a sec.

> I have had reports of this before. Check the /backhand/ page. Is
> everything there marked red? This would be a good sign that the
> moderator isn't running. If it crashes, this is obviously a bug. I
> have been unable to replicate this on any of my installations (running
> for months on end with no problems).

I'm not running backhand at the moment (2-3 crashes a day are too much
for me), but I could put it back in place if you would like to take a
look at it.

>
> --
> Theo Schlossnagle
> Principal Consultant
> OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
> Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
> 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
> 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>
> ---------------------------------------------------------
> -- Presenting at ApacheCon --
> -- November 18th, 2002 --
> -- Las Vegas, Nevada --
> -- Backhand: understanding and building HA/LB clusters --
> -- http://apachecon.com/2002/US/ --
> -- --
> -- Learn all there is to know about high availability --
> -- internet systems and load balancing techniques --
> -- focusing on applications driven by the Apache web --
> -- server! --
> ---------------------------------------------------------
>
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 7, 2002, 1:41 PM

Post #11 of 21 (11657 views)

Permalink

> James Blackwell wrote:
>> No such luck. Any other ideas?
>
> I have similar problems, and I think we found a pattern in our machines.
>
> We receive about 13 million hits a day (small 1x1 images, we do web
> statistics). During the day all our machines work near its max capacity
> (num apache childs ~ max servers). But around 7 PM our traffic suddenly
> decreases, and I start seeing a lot of these messages in our log file
> -umbilica, etc. Recently, I had a misconfiguration (files-max was 8192)
> and every evening my front end machines went south. The pattern is the
> fact that when the backend machines started killing childs due to the
> load decreasing, the front end machines went south.
>
> Of course, I did increase the files-max number and everything is ok now.
> I still see the ocassional error message in my log file, but that's all.

I'm not sure it's files-max. I'm proud of myself when the site gets
2,000 page views a day, and that's including spiders. These crashes also
don't seem to have any solid link to load.

I have some slightly unusual settings in my httpd.conf, but not any that
I think is worth staring crosseyed at... Yes. I noticed that
startservers is higher than maxspares and that keepalivetimeout may be a
bit too small.

MaxKeepAliveRequests 100
KeepAliveTimeout 15

MinSpare = 1
MaxSpare = 10
StartServers = 15
MaxClients = 100
KeepAlive On

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 7, 2002, 1:50 PM

Post #12 of 21 (11659 views)

Permalink

Oh, one more thing. Normally I have two backend boxes, but one of them
is down because of a failed hard drive. My memory may be fuzzy, but it
may have been at about this time that the problems may have started
showing up.

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

leus at epublish

Oct 7, 2002, 3:03 PM

Post #13 of 21 (11654 views)

Permalink

James Blackwell wrote:
> I'm not sure it's files-max. I'm proud of myself when the site gets
> 2,000 page views a day, and that's including spiders. These crashes also
> don't seem to have any solid link to load.
>
> I have some slightly unusual settings in my httpd.conf, but not any that
> I think is worth staring crosseyed at... Yes. I noticed that
> startservers is higher than maxspares and that keepalivetimeout may be a
> bit too small.
>
> MaxKeepAliveRequests 100
> KeepAliveTimeout 15
>
> MinSpare = 1
> MaxSpare = 10
> StartServers = 15
> MaxClients = 100
> KeepAlive On
>

Even if the numbers are different, I would venture to say backhand isn't
cleaning up its connections when a backend machine start killing childs.
In your case, you start with 15 servers, then kill some of them in order
to stay around your maxspare setting.

I know its weird, but I can't think of another cause. My "solution" was
to modify mod_backhand to use a public instead of private shared mem
segment to store its serverstats array, then use a little program and a
cron job to monitor if the first entry of this array have an age value
over 5 seconds. When this happens, it makes a lot of noise and send
emails to a pager, and just then an operator comes in and restart the
whole apache thing.

Not a pretty sight, but at least it kinda worked before we realized that
our files-max values was screwed.
--
Leonardo Herrera L.
mailto:leus@epublish.cl

Loosing umbilical to operator [ In reply to ]

mattcauth at yahoo

Oct 7, 2002, 3:19 PM

Post #14 of 21 (11637 views)

Permalink

For what it's worth - this problem is exactly why I had to stop using backand a
while back. A very similar thing was going on with my setup, although my crashes
were not nearly as predicable. Same exact behavior with the memory. I figured I was
doing something wrong in the build or configs. on Solaris (still a possibility, of
course) and finally just gave up.

Matt

--- James Blackwell <jblack@linuxguru.net> wrote:
> In lists.backhand.users, you wrote:
> >
> > On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
> >> For the last two months I have had a problem with backhand running
> >> reliably. Though backhand starts up with no problem and runs for hours,
> >> after anywhere from 4 hours to two days, backhand starts serving and
> >> starts placing a gratuitious number of the following error messages into
> >> my apache error log file:
> >>
> >> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
> >> failed to establish umbilical to moderator!
> >
> > Sounds to me like the moderator is crashing. Look in your logs fora
> > httpd segfault or something to that effect immediately before the errors
> > start up.
>
> Let me check the logs... Yes. It appears to me that the moderator is
> crashing.
>
>
> recvmsg errorconnection closed by server
> [Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
> Child 13353 failed to establish umbilical to moderator!
>
> recvmsg errorconnection closed by server
> [Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe (13)
> [Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
> failed to establish umbilical to moderato
>
>
> I've noticed two things. After the moderator has clearly crashed, the
> apache processes are using very, very little memory. an RSS around 332.
>
> However, right at about the time that the moderator crashes or is about
> to crash, apache processes on the front end get huge... RSS shows as
> ranging anywhere from 20 - 40 megabytes.
>
>
> Oh, another unusual thing. because of some timing I do, my average page
> guild time is .56 seconds, and when backhand isn't being used displays
> just about as quickly. But when backhand is involved, though page build
> is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a
> sec.
>
> > I have had reports of this before. Check the /backhand/ page. Is
> > everything there marked red? This would be a good sign that the
> > moderator isn't running. If it crashes, this is obviously a bug. I
> > have been unable to replicate this on any of my installations (running
> > for months on end with no problems).
>
>
> I'm not running backhand at the moment (2-3 crashes a day are too much
> for me), but I could put it back in place if you would like to take a
> look at it.
>
>
>
> >
> > --
> > Theo Schlossnagle
> > Principal Consultant
> > OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
> > Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
> > 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
> > 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
> >
> > ---------------------------------------------------------
> > -- Presenting at ApacheCon --
> > -- November 18th, 2002 --
> > -- Las Vegas, Nevada --
> > -- Backhand: understanding and building HA/LB clusters --
> > -- http://apachecon.com/2002/US/ --
> > -- --
> > -- Learn all there is to know about high availability --
> > -- internet systems and load balancing techniques --
> > -- focusing on applications driven by the Apache web --
> > -- server! --
> > ---------------------------------------------------------
> >
> >
> > _______________________________________________
> > backhand-users mailing list
> > backhand-users@lists.backhand.org
> > http://lists.backhand.org/mailman/listinfo/backhand-users
> >
>
>
> --
> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
> James Blackwell -- Director http://www.linuxguru.net
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users

__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com

Loosing umbilical to operator [ In reply to ]

jesus at omniti

Oct 7, 2002, 3:31 PM

Post #15 of 21 (11656 views)

Permalink

Matt C. wrote:

>For what it's worth - this problem is exactly why I had to stop using backand a
>while back. A very similar thing was going on with my setup, although my crashes
>were not nearly as predicable. Same exact behavior with the memory. I figured I was
>doing something wrong in the build or configs. on Solaris (still a possibility, of
>course) and finally just gave up.
>
>
Did you try setting:

BackhandConnectionPools Off

That should make the moderator much less integral to the request serving
process.

--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
-----------------------------------------------------------

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 7, 2002, 3:33 PM

Post #16 of 21 (11646 views)

Permalink

In lists.backhand.users, you wrote:
> For what it's worth - this problem is exactly why I had to stop using backand a
> while back. A very similar thing was going on with my setup, although my crashes
> were not nearly as predicable. Same exact behavior with the memory. I figured I was
> doing something wrong in the build or configs. on Solaris (still a possibility, of
> course) and finally just gave up.
>
> Matt

Oh, I've used backhand for over a year with no problem whatsover.
Absolutely *love* it.

Something has changed. Back in July I moved across country and had boxes
spread around the country.... One in San Diego, a couple in
Pennsylvania, even a couple in the trunk of my car.

I didn't get backhand going again for a couple months afterword,
complete with both backend boxes (cox and rms) serving to either mercury
or jupiter with no problem.

Then, at about the same time I moved backhand to jupiter and the hard
drive in rms died, possibly somewhere in august.

So one of two possiblities seem to be occuring:

- Backhand gets screwy when there is only one eligible box to serve
(mercury and jupiter were both "dumb frontends")

- Jupiter is somehow not performing appropriately, giving backhand heart
attacks. It's possible, but right up until the moment backhand is in
its death throes,

> --- James Blackwell <jblack@linuxguru.net> wrote:
>> In lists.backhand.users, you wrote:
>> >
>> > On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
>> >> For the last two months I have had a problem with backhand running
>> >> reliably. Though backhand starts up with no problem and runs for hours,
>> >> after anywhere from 4 hours to two days, backhand starts serving and
>> >> starts placing a gratuitious number of the following error messages into
>> >> my apache error log file:
>> >>
>> >> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>> >> failed to establish umbilical to moderator!
>> >
>> > Sounds to me like the moderator is crashing. Look in your logs fora
>> > httpd segfault or something to that effect immediately before the errors
>> > start up.
>>
>> Let me check the logs... Yes. It appears to me that the moderator is
>> crashing.
>>
>>
>> recvmsg errorconnection closed by server
>> [Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
>> Child 13353 failed to establish umbilical to moderator!
>>
>> recvmsg errorconnection closed by server
>> [Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe (13)
>> [Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
>> failed to establish umbilical to moderato
>>
>>
>> I've noticed two things. After the moderator has clearly crashed, the
>> apache processes are using very, very little memory. an RSS around 332.
>>
>> However, right at about the time that the moderator crashes or is about
>> to crash, apache processes on the front end get huge... RSS shows as
>> ranging anywhere from 20 - 40 megabytes.
>>
>>
>> Oh, another unusual thing. because of some timing I do, my average page
>> guild time is .56 seconds, and when backhand isn't being used displays
>> just about as quickly. But when backhand is involved, though page build
>> is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a
>> sec.
>>
>> > I have had reports of this before. Check the /backhand/ page. Is
>> > everything there marked red? This would be a good sign that the
>> > moderator isn't running. If it crashes, this is obviously a bug. I
>> > have been unable to replicate this on any of my installations (running
>> > for months on end with no problems).
>>
>>
>> I'm not running backhand at the moment (2-3 crashes a day are too much
>> for me), but I could put it back in place if you would like to take a
>> look at it.
>>
>>
>>
>> >
>> > --
>> > Theo Schlossnagle
>> > Principal Consultant
>> > OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
>> > Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
>> > 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>> > 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>> >
>> > ---------------------------------------------------------
>> > -- Presenting at ApacheCon --
>> > -- November 18th, 2002 --
>> > -- Las Vegas, Nevada --
>> > -- Backhand: understanding and building HA/LB clusters --
>> > -- http://apachecon.com/2002/US/ --
>> > -- --
>> > -- Learn all there is to know about high availability --
>> > -- internet systems and load balancing techniques --
>> > -- focusing on applications driven by the Apache web --
>> > -- server! --
>> > ---------------------------------------------------------
>> >
>> >
>> > _______________________________________________
>> > backhand-users mailing list
>> > backhand-users@lists.backhand.org
>> > http://lists.backhand.org/mailman/listinfo/backhand-users
>> >
>>
>>
>> --
>> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
>> James Blackwell -- Director http://www.linuxguru.net
>>
>> _______________________________________________
>> backhand-users mailing list
>> backhand-users@lists.backhand.org
>> http://lists.backhand.org/mailman/listinfo/backhand-users
>
>
> __________________________________________________
> Do you Yahoo!?
> Faith Hill - Exclusive Performances, Videos & More
> http://faith.yahoo.com
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

george at omniti

Oct 7, 2002, 3:39 PM

Post #17 of 21 (11643 views)

Permalink

Well, if the backhand moderator is segfaulting, there is a bug. Does
the moderator dump core when it dies? Can you provide a full backtrace
from that core?

James Blackwell wrote:

>In lists.backhand.users, you wrote:
>
>>For what it's worth - this problem is exactly why I had to stop using backand a
>>while back. A very similar thing was going on with my setup, although my crashes
>>were not nearly as predicable. Same exact behavior with the memory. I figured I was
>>doing something wrong in the build or configs. on Solaris (still a possibility, of
>>course) and finally just gave up.
>>
>>Matt
>>
>
>Oh, I've used backhand for over a year with no problem whatsover.
>Absolutely *love* it.
>
>Something has changed. Back in July I moved across country and had boxes
>spread around the country.... One in San Diego, a couple in
>Pennsylvania, even a couple in the trunk of my car.
>
>I didn't get backhand going again for a couple months afterword,
>complete with both backend boxes (cox and rms) serving to either mercury
>or jupiter with no problem.
>
>Then, at about the same time I moved backhand to jupiter and the hard
>drive in rms died, possibly somewhere in august.
>
>So one of two possiblities seem to be occuring:
>
>- Backhand gets screwy when there is only one eligible box to serve
> (mercury and jupiter were both "dumb frontends")
>
>- Jupiter is somehow not performing appropriately, giving backhand heart
> attacks. It's possible, but right up until the moment backhand is in
> its death throes,
>
>
>
>
>>--- James Blackwell <jblack@linuxguru.net> wrote:
>>
>>>In lists.backhand.users, you wrote:
>>>
>>>>On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
>>>>
>>>>>For the last two months I have had a problem with backhand running
>>>>>reliably. Though backhand starts up with no problem and runs for hours,
>>>>>after anywhere from 4 hours to two days, backhand starts serving and
>>>>>starts placing a gratuitious number of the following error messages into
>>>>>my apache error log file:
>>>>>
>>>>>[Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>>>>>failed to establish umbilical to moderator!
>>>>>
>>>>Sounds to me like the moderator is crashing. Look in your logs fora
>>>>httpd segfault or something to that effect immediately before the errors
>>>>start up.
>>>>
>>>Let me check the logs... Yes. It appears to me that the moderator is
>>>crashing.
>>>
>>>
>>>recvmsg errorconnection closed by server
>>>[Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
>>> Child 13353 failed to establish umbilical to moderator!
>>>
>>>recvmsg errorconnection closed by server
>>>[Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe (13)
>>>[Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
>>> failed to establish umbilical to moderato
>>>
>>>
>>>I've noticed two things. After the moderator has clearly crashed, the
>>>apache processes are using very, very little memory. an RSS around 332.
>>>
>>>However, right at about the time that the moderator crashes or is about
>>>to crash, apache processes on the front end get huge... RSS shows as
>>>ranging anywhere from 20 - 40 megabytes.
>>>
>>>
>>>Oh, another unusual thing. because of some timing I do, my average page
>>>guild time is .56 seconds, and when backhand isn't being used displays
>>>just about as quickly. But when backhand is involved, though page build
>>>is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a
>>>sec.
>>>
>>>>I have had reports of this before. Check the /backhand/ page. Is
>>>>everything there marked red? This would be a good sign that the
>>>>moderator isn't running. If it crashes, this is obviously a bug. I
>>>>have been unable to replicate this on any of my installations (running
>>>>for months on end with no problems).
>>>>
>>>
>>>I'm not running backhand at the moment (2-3 crashes a day are too much
>>>for me), but I could put it back in place if you would like to take a
>>>look at it.
>>>
>>>
>>>
>>>>--
>>>>Theo Schlossnagle
>>>>Principal Consultant
>>>>OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
>>>>Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
>>>>1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>>>>2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>>>>
>>>>---------------------------------------------------------
>>>>-- Presenting at ApacheCon --
>>>>-- November 18th, 2002 --
>>>>-- Las Vegas, Nevada --
>>>>-- Backhand: understanding and building HA/LB clusters --
>>>>-- http://apachecon.com/2002/US/ --
>>>>-- --
>>>>-- Learn all there is to know about high availability --
>>>>-- internet systems and load balancing techniques --
>>>>-- focusing on applications driven by the Apache web --
>>>>-- server! --
>>>>---------------------------------------------------------
>>>>
>>>>
>>>>_______________________________________________
>>>>backhand-users mailing list
>>>>backhand-users@lists.backhand.org
>>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>>
>>>
>>>--
>>>GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
>>>James Blackwell -- Director http://www.linuxguru.net
>>>
>>>_______________________________________________
>>>backhand-users mailing list
>>>backhand-users@lists.backhand.org
>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>
>>
>>__________________________________________________
>>Do you Yahoo!?
>>Faith Hill - Exclusive Performances, Videos & More
>>http://faith.yahoo.com
>>
>>_______________________________________________
>>backhand-users mailing list
>>backhand-users@lists.backhand.org
>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>
>
>

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 7, 2002, 4:14 PM

Post #18 of 21 (11659 views)

Permalink

In lists.backhand.users, you wrote:
> Well, if the backhand moderator is segfaulting, there is a bug. Does
> the moderator dump core when it dies? Can you provide a full backtrace
> from that core?

Nope, backhand hasn't been providing a core dump, but Debian sets up
starting servers to not provide coredumps to protect against certain
types of malicious activity. Just to make things even more fun, binaries
are distributed stripped. :(

I'm not quite sure how I can convince apache that it should write a core
file. So I figured I could just attach gdb to the moderator process and
let things run until it cores.

I've never run gdb on a .so before, though I've used gdb with standalone
programs plenty. I compiled up a 66k mod_backhand.so that turns into a
55k mod_backhand.so when I strip it, so I think I have a .so with
symbol information. Yet when I attach to the moderator process I get a
gripe that gdb doesn't have any symbol information. However I am able to
start apache, attach to the moderator and cont real quickly, which
avoids the whole core problem.

But unfortunately I don't know how to get around the symbol information
problem... :(

Once I do though, I can backtrace for you, check variables, everything.

> James Blackwell wrote:
>
>>In lists.backhand.users, you wrote:
>>
>>>For what it's worth - this problem is exactly why I had to stop using backand a
>>>while back. A very similar thing was going on with my setup, although my crashes
>>>were not nearly as predicable. Same exact behavior with the memory. I figured I was
>>>doing something wrong in the build or configs. on Solaris (still a possibility, of
>>>course) and finally just gave up.
>>>
>>>Matt
>>>
>>
>>Oh, I've used backhand for over a year with no problem whatsover.
>>Absolutely *love* it.
>>
>>Something has changed. Back in July I moved across country and had boxes
>>spread around the country.... One in San Diego, a couple in
>>Pennsylvania, even a couple in the trunk of my car.
>>
>>I didn't get backhand going again for a couple months afterword,
>>complete with both backend boxes (cox and rms) serving to either mercury
>>or jupiter with no problem.
>>
>>Then, at about the same time I moved backhand to jupiter and the hard
>>drive in rms died, possibly somewhere in august.
>>
>>So one of two possiblities seem to be occuring:
>>
>>- Backhand gets screwy when there is only one eligible box to serve
>> (mercury and jupiter were both "dumb frontends")
>>
>>- Jupiter is somehow not performing appropriately, giving backhand heart
>> attacks. It's possible, but right up until the moment backhand is in
>> its death throes,
>>
>>
>>
>>
>>>--- James Blackwell <jblack@linuxguru.net> wrote:
>>>
>>>>In lists.backhand.users, you wrote:
>>>>
>>>>>On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
>>>>>
>>>>>>For the last two months I have had a problem with backhand running
>>>>>>reliably. Though backhand starts up with no problem and runs for hours,
>>>>>>after anywhere from 4 hours to two days, backhand starts serving and
>>>>>>starts placing a gratuitious number of the following error messages into
>>>>>>my apache error log file:
>>>>>>
>>>>>>[Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>>>>>>failed to establish umbilical to moderator!
>>>>>>
>>>>>Sounds to me like the moderator is crashing. Look in your logs fora
>>>>>httpd segfault or something to that effect immediately before the errors
>>>>>start up.
>>>>>
>>>>Let me check the logs... Yes. It appears to me that the moderator is
>>>>crashing.
>>>>
>>>>
>>>>recvmsg errorconnection closed by server
>>>>[Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
>>>> Child 13353 failed to establish umbilical to moderator!
>>>>
>>>>recvmsg errorconnection closed by server
>>>>[Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe (13)
>>>>[Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
>>>> failed to establish umbilical to moderato
>>>>
>>>>
>>>>I've noticed two things. After the moderator has clearly crashed, the
>>>>apache processes are using very, very little memory. an RSS around 332.
>>>>
>>>>However, right at about the time that the moderator crashes or is about
>>>>to crash, apache processes on the front end get huge... RSS shows as
>>>>ranging anywhere from 20 - 40 megabytes.
>>>>
>>>>
>>>>Oh, another unusual thing. because of some timing I do, my average page
>>>>guild time is .56 seconds, and when backhand isn't being used displays
>>>>just about as quickly. But when backhand is involved, though page build
>>>>is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a
>>>>sec.
>>>>
>>>>>I have had reports of this before. Check the /backhand/ page. Is
>>>>>everything there marked red? This would be a good sign that the
>>>>>moderator isn't running. If it crashes, this is obviously a bug. I
>>>>>have been unable to replicate this on any of my installations (running
>>>>>for months on end with no problems).
>>>>>
>>>>
>>>>I'm not running backhand at the moment (2-3 crashes a day are too much
>>>>for me), but I could put it back in place if you would like to take a
>>>>look at it.
>>>>
>>>>
>>>>
>>>>>--
>>>>>Theo Schlossnagle
>>>>>Principal Consultant
>>>>>OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
>>>>>Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
>>>>>1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>>>>>2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>>>>>
>>>>>---------------------------------------------------------
>>>>>-- Presenting at ApacheCon --
>>>>>-- November 18th, 2002 --
>>>>>-- Las Vegas, Nevada --
>>>>>-- Backhand: understanding and building HA/LB clusters --
>>>>>-- http://apachecon.com/2002/US/ --
>>>>>-- --
>>>>>-- Learn all there is to know about high availability --
>>>>>-- internet systems and load balancing techniques --
>>>>>-- focusing on applications driven by the Apache web --
>>>>>-- server! --
>>>>>---------------------------------------------------------
>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>backhand-users mailing list
>>>>>backhand-users@lists.backhand.org
>>>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>>>
>>>>
>>>>--
>>>>GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
>>>>James Blackwell -- Director http://www.linuxguru.net
>>>>
>>>>_______________________________________________
>>>>backhand-users mailing list
>>>>backhand-users@lists.backhand.org
>>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>>
>>>
>>>__________________________________________________
>>>Do you Yahoo!?
>>>Faith Hill - Exclusive Performances, Videos & More
>>>http://faith.yahoo.com
>>>
>>>_______________________________________________
>>>backhand-users mailing list
>>>backhand-users@lists.backhand.org
>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>
>>
>>
>
>
>
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

mattcauth at yahoo

Oct 8, 2002, 5:42 AM

Post #19 of 21 (11645 views)

Permalink

Interesting...my setup choice may well have contributed to backhand's unhappiness
with me then:

1 frontend, vanilla apache with SSl, with a VERY short timeout on keepalives.
2 backend mod_perl apache machines doing the heavy stuff.

All on Solaris 8, statically compiled. Backhand would work great for a while, then
my apache in on the front-end machine would spin out of control. I recall I did have
connection pools turned off (although I'll have to confirm this - it's been a long
time). I wanted to achieve the common goals of:

a) Offloading SSL/static content to a fast, vanilla apache in front
b) Do some very simple load distribution to the mod_perl apaches

Backhand is perfect for all this. I know that many people use Backhand on Solaris
with success, which frustrates me; I may do a bit more testing with it if my setup
was (is) flawed. I'll try and setup my original config/architecture and reproduce
the error when I get some time.

Matt

--- James Blackwell <jblack@linuxguru.net> wrote:
> In lists.backhand.users, you wrote:
> > For what it's worth - this problem is exactly why I had to stop using backand a
> > while back. A very similar thing was going on with my setup, although my crashes
> > were not nearly as predicable. Same exact behavior with the memory. I figured I
> was
> > doing something wrong in the build or configs. on Solaris (still a possibility,
> of
> > course) and finally just gave up.
> >
> > Matt
>
> Oh, I've used backhand for over a year with no problem whatsover.
> Absolutely *love* it.
>
> Something has changed. Back in July I moved across country and had boxes
> spread around the country.... One in San Diego, a couple in
> Pennsylvania, even a couple in the trunk of my car.
>
> I didn't get backhand going again for a couple months afterword,
> complete with both backend boxes (cox and rms) serving to either mercury
> or jupiter with no problem.
>
> Then, at about the same time I moved backhand to jupiter and the hard
> drive in rms died, possibly somewhere in august.
>
> So one of two possiblities seem to be occuring:
>
> - Backhand gets screwy when there is only one eligible box to serve
> (mercury and jupiter were both "dumb frontends")
>
> - Jupiter is somehow not performing appropriately, giving backhand heart
> attacks. It's possible, but right up until the moment backhand is in
> its death throes,
>
>
>
>
> > --- James Blackwell <jblack@linuxguru.net> wrote:
> >> In lists.backhand.users, you wrote:
> >> >
> >> > On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
> >> >> For the last two months I have had a problem with backhand running
> >> >> reliably. Though backhand starts up with no problem and runs for hours,
> >> >> after anywhere from 4 hours to two days, backhand starts serving and
> >> >> starts placing a gratuitious number of the following error messages into
> >> >> my apache error log file:
> >> >>
> >> >> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
> >> >> failed to establish umbilical to moderator!
> >> >
> >> > Sounds to me like the moderator is crashing. Look in your logs fora
> >> > httpd segfault or something to that effect immediately before the errors
> >> > start up.
> >>
> >> Let me check the logs... Yes. It appears to me that the moderator is
> >> crashing.
> >>
> >>
> >> recvmsg errorconnection closed by server
> >> [Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
> >> Child 13353 failed to establish umbilical to moderator!
> >>
> >> recvmsg errorconnection closed by server
> >> [Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe
> (13)
> >> [Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
> >> failed to establish umbilical to moderato
> >>
> >>
> >> I've noticed two things. After the moderator has clearly crashed, the
> >> apache processes are using very, very little memory. an RSS around 332.
> >>
> >> However, right at about the time that the moderator crashes or is about
> >> to crash, apache processes on the front end get huge... RSS shows as
> >> ranging anywhere from 20 - 40 megabytes.
> >>
> >>
> >> Oh, another unusual thing. because of some timing I do, my average page
> >> guild time is .56 seconds, and when backhand isn't being used displays
> >> just about as quickly. But when backhand is involved, though page build
> >> is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a
> >> sec.
> >>
> >> > I have had reports of this before. Check the /backhand/ page. Is
> >> > everything there marked red? This would be a good sign that the
> >> > moderator isn't running. If it crashes, this is obviously a bug. I
> >> > have been unable to replicate this on any of my installations (running
> >> > for months on end with no problems).
> >>
> >>
> >> I'm not running backhand at the moment (2-3 crashes a day are too much
> >> for me), but I could put it back in place if you would like to take a
> >> look at it.
> >>
> >>
> >>
> >> >
> >> > --
> >> > Theo Schlossnagle
> >> > Principal Consultant
> >> > OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
> >> > Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
> >> > 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
> >> > 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
> >> >
> >> > ---------------------------------------------------------
> >> > -- Presenting at ApacheCon --
> >> > -- November 18th, 2002 --
> >> > -- Las Vegas, Nevada --
> >> > -- Backhand: understanding and building HA/LB clusters --
> >> > -- http://apachecon.com/2002/US/ --
> >> > -- --
> >> > -- Learn all there is to know about high availability --
> >> > -- internet systems and load balancing techniques --
> >> > -- focusing on applications driven by the Apache web --
> >> > -- server! --
> >> > ---------------------------------------------------------
> >> >
> >> >
> >> > _______________________________________________
> >> > backhand-users mailing list
> >> > backhand-users@lists.backhand.org
> >> > http://lists.backhand.org/mailman/listinfo/backhand-users
> >> >
> >>
> >>
> >> --
> >> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
> >> James Blackwell -- Director http://www.linuxguru.net
> >>
> >> _______________________________________________
> >> backhand-users mailing list
> >> backhand-users@lists.backhand.org
> >> http://lists.backhand.org/mailman/listinfo/backhand-users
> >
> >
> > __________________________________________________
> > Do you Yahoo!?
> > Faith Hill - Exclusive Performances, Videos & More
> > http://faith.yahoo.com
> >
> > _______________________________________________
> > backhand-users mailing list
> > backhand-users@lists.backhand.org
> > http://lists.backhand.org/mailman/listinfo/backhand-users
> >
>
>
> --
> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
> James Blackwell -- Director http://www.linuxguru.net
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users

__________________________________________________
Do you Yahoo!?
Faith Hill - Exclusive Performances, Videos & More
http://faith.yahoo.com

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 8, 2002, 10:24 AM

Post #20 of 21 (11655 views)

Permalink

In lists.backhand.users, you wrote:
> Interesting...my setup choice may well have contributed to backhand's unhappiness
> with me then:
>
> 1 frontend, vanilla apache with SSl, with a VERY short timeout on keepalives.
> 2 backend mod_perl apache machines doing the heavy stuff.
>
> All on Solaris 8, statically compiled. Backhand would work great for a while, then
> my apache in on the front-end machine would spin out of control. I recall I did have
> connection pools turned off (although I'll have to confirm this - it's been a long
> time). I wanted to achieve the common goals of:
>
> a) Offloading SSL/static content to a fast, vanilla apache in front
> b) Do some very simple load distribution to the mod_perl apaches
>
> Backhand is perfect for all this. I know that many people use Backhand on Solaris
> with success, which frustrates me; I may do a bit more testing with it if my setup
> was (is) flawed. I'll try and setup my original config/architecture and reproduce
> the error when I get some time.

Yes. backhand (when working) is ideal for the things I am doing at well.

Until I can get the backhand team the backtrace they want though, I'm
going to have to do without backhand though. Reducing conenctions to
9-16k a second and having several crashes a day is rather too much for
me. LinuxGuru.net might be a backwater site in this day and age but with
a readership of 50k a month I'm letting too many people down if I keep
backhand going.

> --- James Blackwell <jblack@linuxguru.net> wrote:
>> In lists.backhand.users, you wrote:
>> > For what it's worth - this problem is exactly why I had to stop using backand a
>> > while back. A very similar thing was going on with my setup, although my crashes
>> > were not nearly as predicable. Same exact behavior with the memory. I figured I
>> was
>> > doing something wrong in the build or configs. on Solaris (still a possibility,
>> of
>> > course) and finally just gave up.
>> >
>> > Matt
>>
>> Oh, I've used backhand for over a year with no problem whatsover.
>> Absolutely *love* it.
>>
>> Something has changed. Back in July I moved across country and had boxes
>> spread around the country.... One in San Diego, a couple in
>> Pennsylvania, even a couple in the trunk of my car.
>>
>> I didn't get backhand going again for a couple months afterword,
>> complete with both backend boxes (cox and rms) serving to either mercury
>> or jupiter with no problem.
>>
>> Then, at about the same time I moved backhand to jupiter and the hard
>> drive in rms died, possibly somewhere in august.
>>
>> So one of two possiblities seem to be occuring:
>>
>> - Backhand gets screwy when there is only one eligible box to serve
>> (mercury and jupiter were both "dumb frontends")
>>
>> - Jupiter is somehow not performing appropriately, giving backhand heart
>> attacks. It's possible, but right up until the moment backhand is in
>> its death throes,
>>
>>
>>
>>
>> > --- James Blackwell <jblack@linuxguru.net> wrote:
>> >> In lists.backhand.users, you wrote:
>> >> >
>> >> > On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
>> >> >> For the last two months I have had a problem with backhand running
>> >> >> reliably. Though backhand starts up with no problem and runs for hours,
>> >> >> after anywhere from 4 hours to two days, backhand starts serving and
>> >> >> starts placing a gratuitious number of the following error messages into
>> >> >> my apache error log file:
>> >> >>
>> >> >> [Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>> >> >> failed to establish umbilical to moderator!
>> >> >
>> >> > Sounds to me like the moderator is crashing. Look in your logs fora
>> >> > httpd segfault or something to that effect immediately before the errors
>> >> > start up.
>> >>
>> >> Let me check the logs... Yes. It appears to me that the moderator is
>> >> crashing.
>> >>
>> >>
>> >> recvmsg errorconnection closed by server
>> >> [Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
>> >> Child 13353 failed to establish umbilical to moderator!
>> >>
>> >> recvmsg errorconnection closed by server
>> >> [Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe
>> (13)
>> >> [Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
>> >> failed to establish umbilical to moderato
>> >>
>> >>
>> >> I've noticed two things. After the moderator has clearly crashed, the
>> >> apache processes are using very, very little memory. an RSS around 332.
>> >>
>> >> However, right at about the time that the moderator crashes or is about
>> >> to crash, apache processes on the front end get huge... RSS shows as
>> >> ranging anywhere from 20 - 40 megabytes.
>> >>
>> >>
>> >> Oh, another unusual thing. because of some timing I do, my average page
>> >> guild time is .56 seconds, and when backhand isn't being used displays
>> >> just about as quickly. But when backhand is involved, though page build
>> >> is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a
>> >> sec.
>> >>
>> >> > I have had reports of this before. Check the /backhand/ page. Is
>> >> > everything there marked red? This would be a good sign that the
>> >> > moderator isn't running. If it crashes, this is obviously a bug. I
>> >> > have been unable to replicate this on any of my installations (running
>> >> > for months on end with no problems).
>> >>
>> >>
>> >> I'm not running backhand at the moment (2-3 crashes a day are too much
>> >> for me), but I could put it back in place if you would like to take a
>> >> look at it.
>> >>
>> >>
>> >>
>> >> >
>> >> > --
>> >> > Theo Schlossnagle
>> >> > Principal Consultant
>> >> > OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
>> >> > Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
>> >> > 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>> >> > 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>> >> >
>> >> > ---------------------------------------------------------
>> >> > -- Presenting at ApacheCon --
>> >> > -- November 18th, 2002 --
>> >> > -- Las Vegas, Nevada --
>> >> > -- Backhand: understanding and building HA/LB clusters --
>> >> > -- http://apachecon.com/2002/US/ --
>> >> > -- --
>> >> > -- Learn all there is to know about high availability --
>> >> > -- internet systems and load balancing techniques --
>> >> > -- focusing on applications driven by the Apache web --
>> >> > -- server! --
>> >> > ---------------------------------------------------------
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > backhand-users mailing list
>> >> > backhand-users@lists.backhand.org
>> >> > http://lists.backhand.org/mailman/listinfo/backhand-users
>> >> >
>> >>
>> >>
>> >> --
>> >> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
>> >> James Blackwell -- Director http://www.linuxguru.net
>> >>
>> >> _______________________________________________
>> >> backhand-users mailing list
>> >> backhand-users@lists.backhand.org
>> >> http://lists.backhand.org/mailman/listinfo/backhand-users
>> >
>> >
>> > __________________________________________________
>> > Do you Yahoo!?
>> > Faith Hill - Exclusive Performances, Videos & More
>> > http://faith.yahoo.com
>> >
>> > _______________________________________________
>> > backhand-users mailing list
>> > backhand-users@lists.backhand.org
>> > http://lists.backhand.org/mailman/listinfo/backhand-users
>> >
>>
>>
>> --
>> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
>> James Blackwell -- Director http://www.linuxguru.net
>>
>> _______________________________________________
>> backhand-users mailing list
>> backhand-users@lists.backhand.org
>> http://lists.backhand.org/mailman/listinfo/backhand-users
>
>
> __________________________________________________
> Do you Yahoo!?
> Faith Hill - Exclusive Performances, Videos & More
> http://faith.yahoo.com
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net

Loosing umbilical to operator [ In reply to ]

jblack at linuxguru

Oct 8, 2002, 10:42 AM

Post #21 of 21 (11653 views)

Permalink

> I'm not quite sure how I can convince apache that it should write a core
> file. So I figured I could just attach gdb to the moderator process and
> let things run until it cores.

I ended up not finding a way to do this. Debian hides this away pretty
darn well. I'm not sure it's pertinant anyways, because I was attaching
gdb to the moderator and was able to wait for crashes.

> I've never run gdb on a .so before, though I've used gdb with standalone
> programs plenty. I compiled up a 66k mod_backhand.so that turns into a
> 55k mod_backhand.so when I strip it, so I think I have a .so with
> symbol information. Yet when I attach to the moderator process I get a
> gripe that gdb doesn't have any symbol information. However I am able to
> start apache, attach to the moderator and cont real quickly, which
> avoids the whole core problem.

This is a bit more serious and I'm at a loss of how to get around this
as well. I can attach to the moderator with gdb and cont until it does
crash with no problem, but I just can't seem to find the way that I'm
supposed to give gdb the symbol information it wants. I even went so far
as to apxs mod_backhand and put that .so in the tree, used the file
command with gdb and pointed it at the unstripped mod_backhand.so (even
the mod_backhand.o just in case)... gdb still insists that it doesn't
have any symbol information.

No symbol information, no useful backtrace.... no useful backtrace, no
useful bug report .... no useful bug report, no reasonable hope for a
fix.

Drats.

Guys, I've really appreciated the effort you went through trying to
help, but I'm guessing that I'll have to do without backhand for awhile.

That's ok... I'm sure that some day this will all come to light. Some
day someone browsing the source tree will add two lines of code in a fit
of inspiration that will clear my unusual problem right up. :)

As an idle thought, would a couple well placed ap_log_errors do anything
for enlightenment? Where would you suggest I put them?

> But unfortunately I don't know how to get around the symbol information
> problem... :(
>
> Once I do though, I can backtrace for you, check variables, everything.

>
>> James Blackwell wrote:
>>
>>>In lists.backhand.users, you wrote:
>>>
>>>>For what it's worth - this problem is exactly why I had to stop using backand a
>>>>while back. A very similar thing was going on with my setup, although my crashes
>>>>were not nearly as predicable. Same exact behavior with the memory. I figured I was
>>>>doing something wrong in the build or configs. on Solaris (still a possibility, of
>>>>course) and finally just gave up.
>>>>
>>>>Matt
>>>>
>>>
>>>Oh, I've used backhand for over a year with no problem whatsover.
>>>Absolutely *love* it.
>>>
>>>Something has changed. Back in July I moved across country and had boxes
>>>spread around the country.... One in San Diego, a couple in
>>>Pennsylvania, even a couple in the trunk of my car.
>>>
>>>I didn't get backhand going again for a couple months afterword,
>>>complete with both backend boxes (cox and rms) serving to either mercury
>>>or jupiter with no problem.
>>>
>>>Then, at about the same time I moved backhand to jupiter and the hard
>>>drive in rms died, possibly somewhere in august.
>>>
>>>So one of two possiblities seem to be occuring:
>>>
>>>- Backhand gets screwy when there is only one eligible box to serve
>>> (mercury and jupiter were both "dumb frontends")
>>>
>>>- Jupiter is somehow not performing appropriately, giving backhand heart
>>> attacks. It's possible, but right up until the moment backhand is in
>>> its death throes,
>>>
>>>
>>>
>>>
>>>>--- James Blackwell <jblack@linuxguru.net> wrote:
>>>>
>>>>>In lists.backhand.users, you wrote:
>>>>>
>>>>>>On Sunday, October 6, 2002, at 09:07 , James Blackwell wrote:
>>>>>>
>>>>>>>For the last two months I have had a problem with backhand running
>>>>>>>reliably. Though backhand starts up with no problem and runs for hours,
>>>>>>>after anywhere from 4 hours to two days, backhand starts serving and
>>>>>>>starts placing a gratuitious number of the following error messages into
>>>>>>>my apache error log file:
>>>>>>>
>>>>>>>[Fri Oct 4 21:49:12 2002] [error] (111)Connection refused: Child 6079
>>>>>>>failed to establish umbilical to moderator!
>>>>>>>
>>>>>>Sounds to me like the moderator is crashing. Look in your logs fora
>>>>>>httpd segfault or something to that effect immediately before the errors
>>>>>>start up.
>>>>>>
>>>>>Let me check the logs... Yes. It appears to me that the moderator is
>>>>>crashing.
>>>>>
>>>>>
>>>>>recvmsg errorconnection closed by server
>>>>>[Mon Oct 7 03:41:18 2002] [error] (111)Connection refused:
>>>>> Child 13353 failed to establish umbilical to moderator!
>>>>>
>>>>>recvmsg errorconnection closed by server
>>>>>[Mon Oct 7 12:55:36 2002] [notice] child pid 25836 exit signal Broken pipe (13)
>>>>>[Mon Oct 7 13:02:19 2002] [error] (111)Connection refused: Child 25950
>>>>> failed to establish umbilical to moderato
>>>>>
>>>>>
>>>>>I've noticed two things. After the moderator has clearly crashed, the
>>>>>apache processes are using very, very little memory. an RSS around 332.
>>>>>
>>>>>However, right at about the time that the moderator crashes or is about
>>>>>to crash, apache processes on the front end get huge... RSS shows as
>>>>>ranging anywhere from 20 - 40 megabytes.
>>>>>
>>>>>
>>>>>Oh, another unusual thing. because of some timing I do, my average page
>>>>>guild time is .56 seconds, and when backhand isn't being used displays
>>>>>just about as quickly. But when backhand is involved, though page build
>>>>>is still .56 seconds, the page (img srcs, etc) each only downloaded at 9-15k a
>>>>>sec.
>>>>>
>>>>>>I have had reports of this before. Check the /backhand/ page. Is
>>>>>>everything there marked red? This would be a good sign that the
>>>>>>moderator isn't running. If it crashes, this is obviously a bug. I
>>>>>>have been unable to replicate this on any of my installations (running
>>>>>>for months on end with no problems).
>>>>>>
>>>>>
>>>>>I'm not running backhand at the moment (2-3 crashes a day are too much
>>>>>for me), but I could put it back in place if you would like to take a
>>>>>look at it.
>>>>>
>>>>>
>>>>>
>>>>>>--
>>>>>>Theo Schlossnagle
>>>>>>Principal Consultant
>>>>>>OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
>>>>>>Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
>>>>>>1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>>>>>>2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>>>>>>
>>>>>>---------------------------------------------------------
>>>>>>-- Presenting at ApacheCon --
>>>>>>-- November 18th, 2002 --
>>>>>>-- Las Vegas, Nevada --
>>>>>>-- Backhand: understanding and building HA/LB clusters --
>>>>>>-- http://apachecon.com/2002/US/ --
>>>>>>-- --
>>>>>>-- Learn all there is to know about high availability --
>>>>>>-- internet systems and load balancing techniques --
>>>>>>-- focusing on applications driven by the Apache web --
>>>>>>-- server! --
>>>>>>---------------------------------------------------------
>>>>>>
>>>>>>
>>>>>>_______________________________________________
>>>>>>backhand-users mailing list
>>>>>>backhand-users@lists.backhand.org
>>>>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>>>>
>>>>>
>>>>>--
>>>>>GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
>>>>>James Blackwell -- Director http://www.linuxguru.net
>>>>>
>>>>>_______________________________________________
>>>>>backhand-users mailing list
>>>>>backhand-users@lists.backhand.org
>>>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>>>
>>>>
>>>>__________________________________________________
>>>>Do you Yahoo!?
>>>>Faith Hill - Exclusive Performances, Videos & More
>>>>http://faith.yahoo.com
>>>>
>>>>_______________________________________________
>>>>backhand-users mailing list
>>>>backhand-users@lists.backhand.org
>>>>http://lists.backhand.org/mailman/listinfo/backhand-users
>>>>
>>>
>>>
>>
>>
>>
>>
>> _______________________________________________
>> backhand-users mailing list
>> backhand-users@lists.backhand.org
>> http://lists.backhand.org/mailman/listinfo/backhand-users
>>
>
>
> --
> GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
> James Blackwell -- Director http://www.linuxguru.net
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>

--
GnuPG fingerprint AAE4 8C76 58DA 5902 761D 247A 8A55 DA73 0635 7400
James Blackwell -- Director http://www.linuxguru.net