Mailing List Archive

Hi,

I think that (weighted) random selection will yield better results than round robin. Also, because the back-end servers are not Apache, they will not be able to "play" the Backhand game, so you can only use redirect to them from the Backhand front end.

This architecture could be more useful if Backhand monitoring part could be run on the back-end servers as well (for OS resource reporting), which is not the way it is written now. In such a case, you could employ a smarter decision strategy. Theo: Any plans to separate the monitoring from the Apache module ?

Cheers,

:) Yair. http://www.cs.jhu.edu/~yairamir

Kevin,

Yair hit this right on the button. mod_backhand is designed to redirect
to mod_backhand servers. Everyone has to *know* about one another.
This is done by UDP (broad|multi)cast. It would be fairly trivial to
write a process that broadcasts systems resource information on the same
address. If you do this, then each of the front end machines will
*know* about the backend machines and start redirecting to them. This
could even be done in perl is someone was motivated. The structure to
build is layed out in serverstats.h and the C struct is WELL commented
-- just remember all things that could be in network byte order are.

There are plans to seperate the information publishing and subscribing
system from the mod_bakchand module itself. I would very much like to
use the exact same code base, and not actually remove it from
mod_backhand -- just allow other programs to use the the applicable
pieces as a library (or something like that).

To answer your question, if you have an equal or lesser number of
front-end machines, you could hack together something like that right
now -- but, I doubt this is the situation. A little hint on how to do
it: the MulticastStats directive allows a server to advertise
themselves as someone else (useful if you run in a
lightweight/heavyweight server pair configuration on a single host).
There is nothing stopping you from actually using another machine's IP.
Then force self redirection and choose something like byRandom. If you
really want RR, you could implement it as a loadable candidacy function
in just a few lines of code: treat the incoming list as a circular queue
and keep a static index into that circular queue that you increment
every time (wrapping when necessary.

"Yair Amir,,," wrote:
> I think that (weighted) random selection will yield better results than round robin. Also, because the back-end servers are not Apache, they will not be able to "play" the Backhand game, so you can only use redirect to them from the Backhand front end.
>
> This architecture could be more useful if Backhand monitoring part could be run on the back-end servers as well (for OS resource reporting), which is not the way it is written now. In such a case, you could employ a smarter decision strategy. Theo: Any plans to separate the monitoring from the Apache module ?

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

>>>>> "t" == Theo E Schlossnagle <theos@cnds.jhu.edu> writes:

t> Yair hit this right on the button. mod_backhand is designed to redirect
t> to mod_backhand servers. Everyone has to *know* about one another.

Is it strictly speaking everyone, or just the front end servers which
have to know about the backend?

t> It would be fairly trivial to write a process that broadcasts
t> systems resource information on the same address.

Yep, i was thinking of either a standalone process, or maybe something
using the NSAPI and slotting into our netscape server.

I may hack together the first thing as a feasibility study and then
move on to the NSAPI if it looks good. I'll let you know if i end up
with anything clean enough to be contributed :)

I don't think it will be a problem, but we're on multiple subnets and
i don't think we can use multicast. I think a directed broadcast will
do the trick. From memory, it can't do more than one broad/multicast
since you can only have one statement?

Also, you may want to add a note to the INSTALL or somewhere that it
works on solaris 2.6 and above. I tried a 2.5.1 compile and snprintf
didn't exist back then.

Also II, there's a note in the ChangeLog,

* Added support for building mod_backhand as a static module in Apache.
This will fix as-of-yet untackled problems with having a shared
mod_backhand and shared candidacy functions under Solaris.

can you expand on this, I wasn't sure what it meant? Are you saying
the byHostname.so example won't work?


regards

|<evin

--
Kevin J Walters Morgan Stanley Dean Witter
kevin.walters@msdw.com, kjw@acm.org 750 Seventh Avenue
Tel: 212 762 1752 New York
Fax: 212 762 0174 NY 10019

>>>>> "k" == Kevin J Walters <kevin.walters@msdw.com> writes:

>>>>> "t" == Theo E Schlossnagle <theos@cnds.jhu.edu> writes:

k> Also, you may want to add a note to the INSTALL or somewhere that it
k> works on solaris 2.6 and above. I tried a 2.5.1 compile and snprintf
k> didn't exist back then.

Actually, the code contains only two calls to snprintf. I've converted
them to ap_snprintf and it works on 2.5.1.

Are these an oversight? grep ' snprintf' *.c reveals them...


regards

|<evin

--
Kevin J Walters Morgan Stanley Dean Witter
kevin.walters@msdw.com, kjw@acm.org 750 Seventh Avenue
Tel: 212 762 1752 New York
Fax: 212 762 0174 NY 10019

Ummm.... Oops :)

There are only two... "How did those get in there?" Completely an
oversight. They have now been changed in CVS..

Thanks!

Kevin J Walters wrote:
> k> Also, you may want to add a note to the INSTALL or somewhere that it
> k> works on solaris 2.6 and above. I tried a 2.5.1 compile and snprintf
> k> didn't exist back then.
>
> Actually, the code contains only two calls to snprintf. I've converted
> them to ap_snprintf and it works on 2.5.1.
>
> Are these an oversight? grep ' snprintf' *.c reveals them...

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

Kevin J Walters wrote:
> t> Yair hit this right on the button. mod_backhand is designed to redirect
> t> to mod_backhand servers. Everyone has to *know* about one another.
>
> Is it strictly speaking everyone, or just the front end servers which
> have to know about the backend?

Backhand doesn't try to promote or demote the use of "front-end"
servers. So, with that said, just the front-end servers (depending on
your intentions). Any server that you want to proxying or redirect
connections FROM needs to know about all of the machines it will be
forwarding to. So those are who needs to know and the others that want
to "play the game" need to announce (or be announced for).

> t> It would be fairly trivial to write a process that broadcasts
> t> systems resource information on the same address.
> Yep, i was thinking of either a standalone process, or maybe something
> using the NSAPI and slotting into our netscape server.

I am in the process of wrapping a standalone process around the resource
information collection code already in mod_backhand. This will allow
you to compile it on any platform that mod_backhand supports and
announce statistics. (like a Netscape or other http server)

> I may hack together the first thing as a feasibility study and then
> move on to the NSAPI if it looks good. I'll let you know if i end up
> with anything clean enough to be contributed :)

Excellent... Contributions are always welcome.

> I don't think it will be a problem, but we're on multiple subnets and
> i don't think we can use multicast. I think a directed broadcast will
> do the trick. From memory, it can't do more than one broad/multicast
> since you can only have one statement?

Multicast is a Godd Thing(TM). It won't occupy machines that aren't
partiticpating (assuming your switch understands IP multicast -- most
good ones do). With that said, directed broadcasts should work fine. I
do believe you can have multiple statements. This isn't thoroughly
tested, but it was built to support this. Just add multiple
MulticastStats statements in your httpd.conf. It will listen on very
port that it broadcasts on.

> Also, you may want to add a note to the INSTALL or somewhere that it
> works on solaris 2.6 and above. I tried a 2.5.1 compile and snprintf
> didn't exist back then.

Fixed that... Thanks.

> Also II, there's a note in the ChangeLog,
> * Added support for building mod_backhand as a static module in Apache.
> This will fix as-of-yet untackled problems with having a shared
> mod_backhand and shared candidacy functions under Solaris.
> can you expand on this, I wasn't sure what it meant? Are you saying
> the byHostname.so example won't work?

There are *issues* on Solaris with dynamically loaded objects. I
investigated this for a day or two, but got nowhere. If you run a
program, P, and P loads A, then A loads B, then P unloads A and reloads
A, then when A tried to load B again things all go screwy. This is, as
far as I can tell, what is going on. Apache loads all the modules once
to check the conf and then does something strange and it breaks *sub*
loaded DSO's.

So, the solution: compile mod_backhand *not* as a shared object, but
rather statically into Apache. Then, the DSO loading of candidacy
functions will work like a charm. This is unfortunate, but I have yet
found a "correct" solution.

So, the example, strictly speaking, works on Solaris ONLY if you compile
mod_backhand statically into Apache. If you find evidence to the
contrary, please let me know. That would be very valuable.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

>>>>> "t" == Theo E Schlossnagle <theos@cnds.jhu.edu> writes:


I have written, well bluntly hacked away at back_util.c until it
compiled/produced some output, to produce something similar

duchamp$ \
> ./dummy2 $(hostname) $(hostname) 8000 144.14.151.127:4445
Welcome to dummy host=duchamp service=duchamp:8000

It's of sufficient quality to allow me to test and probably very
little else.

t> Multicast is a Godd Thing(TM).

Agreed, but we don't always have it enabled on our routers for a
variety of reasons, some of which are valid. I'm going to test with
directed broadcasts and move on later.

t> So, the example, strictly speaking, works on Solaris ONLY if you compile
t> mod_backhand statically into Apache. If you find evidence to the
t> contrary, please let me know. That would be very valuable.

I'll have a look at this. I got a seg fault earlier when i tried it.

BTW, random note I, I think the byLoad function is not working as
documented. I think the documented version sounds good, but the
implementation looks like it works in 1/1000th's and adds, rather than
subtracts the value?

Random note II, i've not yet got to the prediction stuff but it would
be good if you could specify the fiddle factor.

Ah, and III, almost forgot, the byLoad/byCost and maybe byCPU take no
consideration of ncpu. Seems wrong? I think you'll get away with it if
all you backend servers are the same, but in a mixed
environment... You could mix a simple smp efficiency factor (0.9x)
into the decision to allow for imperfect scalability too?


regards

|<evin

--
Kevin J Walters Morgan Stanley Dean Witter
kevin.walters@msdw.com, kjw@acm.org 750 Seventh Avenue
Tel: 212 762 1752 New York
Fax: 212 762 0174 NY 10019

Kevin J Walters wrote:
> t> Multicast is a Godd Thing(TM).
> Agreed, but we don't always have it enabled on our routers for a
> variety of reasons, some of which are valid. I'm going to test with
> directed broadcasts and move on later.

Agreed.

> BTW, random note I, I think the byLoad function is not working as
> documented. I think the documented version sounds good, but the
> implementation looks like it works in 1/1000th's and adds, rather than
> subtracts the value?

This is was brought up on the users site. If you want a bias of 1 for
your own machine, you actually need to do byLoad -1000 (the sign is
wrong too). This has been fixed in CVS and will be incorportated into
the next version.

> Random note II, i've not yet got to the prediction stuff but it would
> be good if you could specify the fiddle factor.

Hmmm. There is no fiddle factor (it doesn't take an argument). The
amount of load it adds is based on the amount of time an average request
takes on a given machine. So, it assumes that the request takes 100% of
the CPU for the time that it runs. This isn't true, but that is OK
beacuse all servers act this way. If it keeps the request local, it
only adds half of the value.

> Ah, and III, almost forgot, the byLoad/byCost and maybe byCPU take no
> consideration of ncpu. Seems wrong? I think you'll get away with it if
> all you backend servers are the same, but in a mixed
> environment... You could mix a simple smp efficiency factor (0.9x)
> into the decision to allow for imperfect scalability too.

The Load is already adjusted for ncpu, so byLoad and byCost both
inherently take into account ncpu. byCPU does depending on how a
platform represents itself (Linux and Solaris should both be ok). We do
not take into account smp effeciency. Perhaps we will integrate that
in.

Thanks for you input.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

Hi,

One last question, is there any scope for caching content with
mod_backhand?

If the primary use is for farming out heavy cgi/servlet requests then
i suspect this wasn't a (major) consideration in the design?

I get the impression the in-built caching stuff is intimately tied
with the internal proxy code so can't be used.


regards

|<evin

--
Kevin J Walters Morgan Stanley Dean Witter
kevin.walters@msdw.com, kjw@acm.org 750 Seventh Avenue
Tel: 212 762 1752 New York
Fax: 212 762 0174 NY 10019

Kevin J Walters wrote:
> One last question, is there any scope for caching content with
> mod_backhand?

Hmmm, not with mod_backhand directly.

> If the primary use is for farming out heavy cgi/servlet requests then
> i suspect this wasn't a (major) consideration in the design?

Actually this was a major concern, but still cching was not built into
the architecture.

> I get the impression the in-built caching stuff is intimately tied
> with the internal proxy code so can't be used.

I am not sure whether the caching code is intimately tied with
mod_proxy. However, there was motiviation (never went anywhere) to
integrate the logic of mod_backhand into squid. It actually shouldn't
be that hard.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7