Mailing List Archive

George Nguyen wrote:
>
> Sorry if this is a bit naieve, but there's a real lack of documentation and
> discussion about mod_backhand on the web. I have started using it and it
> seemed pretty good, but I've run into some problems and questions.
>
> Here's my configuration. I have three machines, www0, www1 and www2. I'm
> testing mod_backhand on all these servers. mod_backhand v1.1.0 compiled
> without errors into Apache v1.3.12 with PHP.
>
> Each of the machines has an Apache configuration that looks like this:
>
> <Location /backhand/>
> Order deny,allow
> Deny from all
> Allow from x.x.x.0/24
>
> SetHandler backhand-handler
> </Location>
>
> UnixSocketDir /var/local/backhand

Make sure that /var/local/backhand exists and is owned by your apache user
(mine is /opt/apache/backhand/ and my directory looks like this:
drwx------ 2 nobody root 4096 Sep 8 12:19 /opt/apache/backhand

> MulticastStats x.x.x.63:4445

You are accepting stats from a /24, but you are broacasting on a /26 or less?
Is this intentional? It could be correct, but are you sure? Are you sure the
broadcast address of you machines is x.x.x.63 and if so, you may want to
consider matching you Acceptstats to your netblock. If that is part of a /28
netblock you can say x.x.x.48/28. Note that the AcceptStat you have will not
break things.

> AcceptStats x.x.x.0/24
>
> <VirtualHost www0.domain.com>
> <Directory /home/webroot>
> Backhand byAge
> Backhand byRandom
> Backhand byLoad
> </Directory>
> </VirtualHost>

the byRandom in this setup doesn't do anything. You are reordering randomly
and then choosing the least loaded server. The least loaded server will not
change after a radnom ordering. Use either one or the other, but not both
(unless you put a byLogWindow in betweem). Your cluster is a little small to
use the byRandom -> byLogWindow, so I would just stick with:
Backhand byAge
Backhand byLoad

> Each machine has an identical setup except the www0 in the virtualhost
> container is switched with whatever machine name it really is, ie. www0,
> www1 or www2.
>
> When all this is said and done, the url http://www0.domain.com/ acts really
> strangely. Everynow and then all requests will be 403'd or permission denied
> by www0. This happens in waves, where the requests are honored normally for
> a while, and then suddenly changes to being refused on permissions, and then
> back to being accepted normally.

That sounds like it services requests correctly as long as it is serving them
, but when it decides to proxy them, it fails. Check your error logs, there
may be a clue in there.

> I've been checking the logs of these machines, and it seems like only www0
> gets loaded and is the only machine servicing requests.

Check the errorlogs, is there anything disturbing in there? You may need to
turn on logging, try:
BackhandLogLevel +dcsnall,+mbcsall

> Also, looking at the backhand status page on www0, it lists all three
> servers www0, www1 and www2. The strange thing is that for all of them, it
> lists 0 ready servers and 0 total servers.

gracefully restart your servers. /path/to/apachectl graceful

> I don't really know where I'm going with this or what help I'm looking for
> exactly. If anyone has run into similar problems, please describe them. If
> anyone wants me to post more detailed information, I can. I just want to
> understand how mod_backhand works so I can decide whether or not we really
> want to use it.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

OK... I did everything that you had said. You were right, it seems that every request that the server decides to hand off to one of the other servers in the cluster gets permission denied.

I changed my backhand settings to byAge, byRandom. Using the test CGI script, I was able to see that any requests which the server decides to do by itself comes through OK. If the server decides to pass off to another backhand server, it is denied.

This is the error log produced by the main server (www0):

[Fri Sep 8 15:42:18 2000] [notice] (Translate <=) = { /backhand/, (null), (null) }
[Fri Sep 8 15:42:18 2000] [notice] (Translate =>) = { /backhand/, (null), (null) }
[Fri Sep 8 15:42:18 2000] [notice] (Type <=) = { (null), /backhand/, (null) }
[Fri Sep 8 15:42:18 2000] [notice] Func executed for (null) [byAge(NULL)] (3 -> 3)
[Fri Sep 8 15:42:18 2000] [notice] Func executed for (null) [byRandom(NULL)] (3 -> 3)
[Fri Sep 8 15:42:18 2000] [notice] All funcs executed -> www1.domain.com
[Fri Sep 8 15:42:18 2000] [notice] (Type =>) = *{ backhand-redirection, backhand:/backhand/, backhand/redirect }
[Fri Sep 8 15:42:18 2000] [notice] (Postread =>) = { Backhanding to 1 }
[Fri Sep 8 15:42:18 2000] [notice] (Postread =>) = { backhand:/backhand/, backhand-redirection, (null) }
[Fri Sep 8 15:42:18 2000] [notice] (Translate <=) = { backhand:/backhand/, backhand-redirection, (null) }

The other backhanded servers (www1,www2) don't seem to show any traces of any transactions at all between them and the main server (www0).

How do I interpret the above log files? Any ideas of what's going on?

Thanks...



------Original Message------
From: "Theo E. Schlossnagle" <theos@cnds.jhu.edu>
To: George Nguyen <gnguyen@email.com>
Sent: September 8, 2000 5:42:05 PM GMT
Subject: Re: [mod_backhand-users] Configuration problems...


George Nguyen wrote:
>
> Sorry if this is a bit naieve, but there's a real lack of documentation and
> discussion about mod_backhand on the web. I have started using it and it
> seemed pretty good, but I've run into some problems and questions.
>
> Here's my configuration. I have three machines, www0, www1 and www2. I'm
> testing mod_backhand on all these servers. mod_backhand v1.1.0 compiled
> without errors into Apache v1.3.12 with PHP.
>
> Each of the machines has an Apache configuration that looks like this:
>
> <Location /backhand/>
> Order deny,allow
> Deny from all
> Allow from x.x.x.0/24
>
> SetHandler backhand-handler
> </Location>
>
> UnixSocketDir /var/local/backhand

Make sure that /var/local/backhand exists and is owned by your apache user
(mine is /opt/apache/backhand/ and my directory looks like this:
drwx------ 2 nobody root 4096 Sep 8 12:19 /opt/apache/backhand

> MulticastStats x.x.x.63:4445

You are accepting stats from a /24, but you are broacasting on a /26 or less?
Is this intentional? It could be correct, but are you sure? Are you sure the
broadcast address of you machines is x.x.x.63 and if so, you may want to
consider matching you Acceptstats to your netblock. If that is part of a /28
netblock you can say x.x.x.48/28. Note that the AcceptStat you have will not
break things.

> AcceptStats x.x.x.0/24
>
> <VirtualHost www0.domain.com>
> <Directory /home/webroot>
> Backhand byAge
> Backhand byRandom
> Backhand byLoad
> </Directory>
> </VirtualHost>

the byRandom in this setup doesn't do anything. You are reordering randomly
and then choosing the least loaded server. The least loaded server will not
change after a radnom ordering. Use either one or the other, but not both
(unless you put a byLogWindow in betweem). Your cluster is a little small to
use the byRandom -> byLogWindow, so I would just stick with:
Backhand byAge
Backhand byLoad

> Each machine has an identical setup except the www0 in the virtualhost
> container is switched with whatever machine name it really is, ie. www0,
> www1 or www2.
>
> When all this is said and done, the url http://www0.domain.com/ acts really
> strangely. Everynow and then all requests will be 403'd or permission denied
> by www0. This happens in waves, where the requests are honored normally for
> a while, and then suddenly changes to being refused on permissions, and then
> back to being accepted normally.

That sounds like it services requests correctly as long as it is serving them
, but when it decides to proxy them, it fails. Check your error logs, there
may be a clue in there.

> I've been checking the logs of these machines, and it seems like only www0
> gets loaded and is the only machine servicing requests.

Check the errorlogs, is there anything disturbing in there? You may need to
turn on logging, try:
BackhandLogLevel +dcsnall,+mbcsall

> Also, looking at the backhand status page on www0, it lists all three
> servers www0, www1 and www2. The strange thing is that for all of them, it
> lists 0 ready servers and 0 total servers.

gracefully restart your servers. /path/to/apachectl graceful

> I don't really know where I'm going with this or what help I'm looking for
> exactly. If anyone has run into similar problems, please describe them. If
> anyone wants me to post more detailed information, I can. I just want to
> understand how mod_backhand works so I can decide whether or not we really
> want to use it.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------
FREE! The World's Best Email Address @email.com
Reserve your name now at http://www.email.com

George Nguyen wrote:
> I changed my backhand settings to byAge, byRandom. Using the test CGI script, I was able to see that any requests which the server decides to do by itself comes through OK. If the server decides to pass off to another backhand server, it is denied.
>
> This is the error log produced by the main server (www0):
>
> [Fri Sep 8 15:42:18 2000] [notice] (Translate <=) = { /backhand/, (null), (null) }
> [Fri Sep 8 15:42:18 2000] [notice] (Translate =>) = { /backhand/, (null), (null) }
> [Fri Sep 8 15:42:18 2000] [notice] (Type <=) = { (null), /backhand/, (null) }
> [Fri Sep 8 15:42:18 2000] [notice] Func executed for (null) [byAge(NULL)] (3 -> 3)
> [Fri Sep 8 15:42:18 2000] [notice] Func executed for (null) [byRandom(NULL)] (3 -> 3)
> [Fri Sep 8 15:42:18 2000] [notice] All funcs executed -> www1.domain.com
> [Fri Sep 8 15:42:18 2000] [notice] (Type =>) = *{ backhand-redirection, backhand:/backhand/, backhand/redirect }
> [Fri Sep 8 15:42:18 2000] [notice] (Postread =>) = { Backhanding to 1 }
> [Fri Sep 8 15:42:18 2000] [notice] (Postread =>) = { backhand:/backhand/, backhand-redirection, (null) }
> [Fri Sep 8 15:42:18 2000] [notice] (Translate <=) = { backhand:/backhand/, backhand-redirection, (null) }
>
> The other backhanded servers (www1,www2) don't seem to show any traces of any transactions at all between them and the main server (www0).
>
> How do I interpret the above log files? Any ideas of what's going on?

the above logs look fine. Basically, the important lines are in the middle:
# This line says 3 servers went in the byAge candidacy and 3 came out
# So all of your servers are alive
Func executed for (null) [byAge(NULL)] (3-> 3)
# This line says 3 servers went in the byRandom and 3 came out
# that is right (they should just be in a different order (perhaps)
Func executed for (null) [byRandom(NULL)] (3 -> 3)
# This line says that the server chosen was www1.domain.com
All funcs executed -> www1.domain.com

What we are missing is the MBCSP (mod_backhand control system protocol)
debuggin messages. If you set BackhandLogLevel +mbcsall they should show up.
If you did that before, try putting it on its own line. So, if you want dcsn
(decisions) and mbcs verbose logging put:
BackhandLogLevel +mbcsall
BackhandLogLevel +dcsnall

Since we see the decision part is working correctly, you don't need to include
that anymore. It seems like something is going wrong with the IPC
(inter-process communication).

Also, include the output of the backhand status page. That may give a clue.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

Thanks much for helping me solve this mystery...

I did have the BackhandLogLevel +mbcsall set, but it doesn't produce any messages on its own. Only BackhandLogLevel +dcsnall
seems to produce any debugging messages. Perhaps the two are not logging to the same file?

What can cause problems with the IPC? I have some restrictive hosts.allow/hosts.deny settings, but I don't think those would affect Apache, since Apache is running as a standalone server.

I am still a little confused about the process and means mod_backhand takes to connect to the other slave servers and request pages from them. Does it use Apache's internal proxy module? Does it matter that my Backhand directives are inside VirtualHost containers which are set as different servers (www0, www1, www2)?

As you can see, I have many questions, and I'm not even sure how to phrase them all.

Here's the output from the backhand status page:

Entry Hostname Age Address Total Mem Avail Mem # ready servers/
# total servers ~ms/req [#req] Arriba # CPUs Load/HWM CPU Idle
0 www0.domain.com 0 x.x.x.44:80 994394112 868570304 0/0 0 [0] 2861515 2 0.080000/1 0.996000
1 www2.domain.com 1 x.x.x.46:80 529305600 481584320 0/0 0 [0] 2856053 2 0.000000/1 0.996000
2 www1.domain.com 1 x.x.x.45:80 529264640 480347328 0/0 0 [0] 2852205 2 0.000000/1 0.993000

Thanks,
George

------Original Message------
From: "Theo E. Schlossnagle" <theos@cnds.jhu.edu>
To: George Nguyen <gnguyen@email.com>
Sent: September 9, 2000 12:50:49 AM GMT
Subject: Re: [mod_backhand-users] Configuration problems...


George Nguyen wrote:
> I changed my backhand settings to byAge, byRandom. Using the test CGI script, I was able to see that any requests which the server decides to do by itself comes through OK. If the server decides to pass off to another backhand server, it is denied.
>
> This is the error log produced by the main server (www0):
>
> [Fri Sep 8 15:42:18 2000] [notice] (Translate <=) = { /backhand/, (null), (null) }
> [Fri Sep 8 15:42:18 2000] [notice] (Translate =>) = { /backhand/, (null), (null) }
> [Fri Sep 8 15:42:18 2000] [notice] (Type <=) = { (null), /backhand/, (null) }
> [Fri Sep 8 15:42:18 2000] [notice] Func executed for (null) [byAge(NULL)] (3 -> 3)
> [Fri Sep 8 15:42:18 2000] [notice] Func executed for (null) [byRandom(NULL)] (3 -> 3)
> [Fri Sep 8 15:42:18 2000] [notice] All funcs executed -> www1.domain.com
> [Fri Sep 8 15:42:18 2000] [notice] (Type =>) = *{ backhand-redirection, backhand:/backhand/, backhand/redirect }
> [Fri Sep 8 15:42:18 2000] [notice] (Postread =>) = { Backhanding to 1 }
> [Fri Sep 8 15:42:18 2000] [notice] (Postread =>) = { backhand:/backhand/, backhand-redirection, (null) }
> [Fri Sep 8 15:42:18 2000] [notice] (Translate <=) = { backhand:/backhand/, backhand-redirection, (null) }
>
> The other backhanded servers (www1,www2) don't seem to show any traces of any transactions at all between them and the main server (www0).
>
> How do I interpret the above log files? Any ideas of what's going on?

the above logs look fine. Basically, the important lines are in the middle:
# This line says 3 servers went in the byAge candidacy and 3 came out
# So all of your servers are alive
Func executed for (null) [byAge(NULL)] (3-> 3)
# This line says 3 servers went in the byRandom and 3 came out
# that is right (they should just be in a different order (perhaps)
Func executed for (null) [byRandom(NULL)] (3 -> 3)
# This line says that the server chosen was www1.domain.com
All funcs executed -> www1.domain.com

What we are missing is the MBCSP (mod_backhand control system protocol)
debuggin messages. If you set BackhandLogLevel +mbcsall they should show up.
If you did that before, try putting it on its own line. So, if you want dcsn
(decisions) and mbcs verbose logging put:
BackhandLogLevel +mbcsall
BackhandLogLevel +dcsnall

Since we see the decision part is working correctly, you don't need to include
that anymore. It seems like something is going wrong with the IPC
(inter-process communication).

Also, include the output of the backhand status page. That may give a clue.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------
FREE! The World's Best Email Address @email.com
Reserve your name now at http://www.email.com