Mailing List Archive

[mod_backhand-users] (no subject)
I am having a problem with mod_backhand. I have two machines running
FreeBSD 4.1 with Apache-1.3.12 with mod_backhand 1.1.0.
I have tested that the configuration works with the default server dishing
up the backhand-test script by balancing with byAge and byCost(in that
order) but when I use it in a virtualhost directive and it is handed off...
It goes to the default root server on the forwarded box. It seems as if
the responding box is ignoring the Host: negotiation. Has anyone else seen
this problem or can give me some pointers to check? I've verified that the
box that responds with the root server instead of the virt host will indeed
serve the actual virthost. This seems to occur with every virtualhost
specified directive I try to balance, yet the balancing of the default root
works perfectly. I also used tcpdump to grab some packets and verify that
indeed Host: is being passed off.. Perhaps it's malformed?

02:58:39.906147 changedservername.domain.net.news > 10.1.1.1.8080: P
1:365(364) ack 1 win 17520 (DF)
(That line identify's the answering server is requesting the backhanded
page from the second server(the one who responds with the default dir)

0x0000 4500 0194 38e3 4000 4006 1a35 d818 1b0d E...8.@.@..5....
0x0010 d818 1b0e 07d9 1f90 cbc7 9b90 fbbf 7b31 ..............{1
0x0020 5018 4470 c158 0000 4745 5420 2f61 7061 P.Dp.X..GET./apa
0x0030 6368 655f 7062 2e67 6966 2048 5454 502f che_pb.gif.HTTP/
0x0040 312e 310a 4163 6365 7074 3a20 2a2f 2a0a 1.1.Accept:.*/*.
0x0050 4163 6365 7074 2d45 6e63 6f64 696e 673a Accept-Encoding:
0x0060 2067 7a69 702c 2064 6566 6c61 7465 0a41 .gzip,.deflate.A
0x0070 6363 6570 742d 4c61 6e67 7561 6765 3a20 ccept-Language:.
0x0080 656e 2d75 730a 436f 6e6e 6563 7469 6f6e en-us.Connection
0x0090 3a20 4b65 6570 2d41 6c69 7665 0a48 6f73 :.Keep-Alive.Hos
0x00a0 743a 2077 7777 2e64 6865 6c74 6f6e 2e77 t:.www.dhelton.b
0x00b0 696e 2e6e 6574 3a38 3038 300a 4966 2d4d bob.com:8080.If-M
0x00c0 6f64 6966 6965 642d 5369 6e63 653a 2057 odified-Since:.W
0x00d0 6564 2c20 3033 204a 756c 2031 3939 3620 ed,.03.Jul.1996.
0x00e0 3036 3a31 383a 3135 2047 4d54 0a49 662d 06:18:15.GMT.If-
0x00f0 4e6f 6e65 2d4d 6174 6368 3a20 2231 6434 None-Match:."1d4
0x0100 3234 392d 3931 362d 3331 6461 3130 6137 249-916-31da10a7
0x0110 220a 5265 6665 7265 723a 2068 7474 703a ".Referer:.http:
0x0120 2f2f 7777 772e 6468 656c 746f 6e2e 7769 //www.dhelton.bo
0x0130 6e2e 6e65 743a 3830 3830 2f0a 5573 6572 b.com:8080/.User
0x0140 2d41 6765 6e74 3a20 4d6f 7a69 6c6c 612f -Agent:.Mozilla/
0x0150 342e 3020 2863 6f6d 7061 7469 626c 653b 4.0.(compatible;
0x0160 204d 5349 4520 352e 353b 2057 696e 646f .MSIE.5.5;.Windo
0x0170 7773 2039 3829 0a42 6163 6b68 616e 6450 ws.98).BackhandP
0x0180 726f 7869 6564 3a20 3234 2e36 2e35 372e roxied:.25.6.52.
0x0190 3930 0a0a 99..


0x0090 shows that the offending server does indeed receive a Host:
request.... Now either my offending server is denying that it exists for some
reason or the server handing it off it malforming the request. Anyone have
some ideas as to what I can do or use to hunt down the real problem here?
[mod_backhand-users] (no subject) [ In reply to ]
Kyle,

I have been running mod_bachand since version aout 1.0.5 with virtualhosts and
have never had a problem. I get the vibe that you are running this in a
single-point proxy configuration. You have one machine to which request
originate and it can "backhand" them to other machines in your cluster, right?

Have you tried connecting directly to one of the back end machines and making
a HTTP request (including Host: header) and see if it returns the info for the
correct virtualhost or if it instead returns the root server info.

If the request made from A -> B has the Host: header in it correctly (as
validated by tcpdump), then it is more or less out of my hands :-) The
proxied request has been modified to include the BackhandProxied: header, so
the remote mod_backhand instance should not attempt to "re"balance the
request.

You could feasibly turn on debugging on the backend machine and see if it is
trying to redirect the requests again, but I don't see how that could be
happenning...

Kyle Stone wrote:
>
> I am having a problem with mod_backhand. I have two machines running
> FreeBSD 4.1 with Apache-1.3.12 with mod_backhand 1.1.0.
> I have tested that the configuration works with the default server dishing
> up the backhand-test script by balancing with byAge and byCost(in that
> order) but when I use it in a virtualhost directive and it is handed off...
> It goes to the default root server on the forwarded box. It seems as if
> the responding box is ignoring the Host: negotiation. Has anyone else seen
> this problem or can give me some pointers to check? I've verified that the
> box that responds with the root server instead of the virt host will indeed
> serve the actual virthost. This seems to occur with every virtualhost
> specified directive I try to balance, yet the balancing of the default root
> works perfectly. I also used tcpdump to grab some packets and verify that
> indeed Host: is being passed off.. Perhaps it's malformed?
> [ ... snip ... ]
> 0x0090 shows that the offending server does indeed receive a Host:
> request.... Now either my offending server is denying that it exists for some
> reason or the server handing it off it malforming the request. Anyone have
> some ideas as to what I can do or use to hunt down the real problem here?
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
[mod_backhand-users] (no subject) [ In reply to ]
When you are running back-hand on a cluster and you wish to disable one of
the machines because, for example the
database server is down on that machine, how would you do that?

Please reply to my email address as I am not subscribed to this list.
[mod_backhand-users] (no subject) [ In reply to ]
This depends tremendously on your set up. If you have all of the machines
externally accessible and you advertise the machine, it is damn hard. :-) An
example is 5 machines running Apache+mod_backhand and you advertise all five
machines through DNS. The problem here is that users will still *originate*
requests to that machine.

If you are using another solution (a HA solution like Arrowpoint, BIG/ip, LVS,
ServerIron, etc.) then when you shut the web server down on that machine,
those products will *know* and stop directing requests to it.

Also, if you have mod_backhand set up in a proxy configuration then clients
will never *originate* requests directly to a web server in the back. This is
were you have a handful of machines running mod_bakchand that are not publicly
advertised and one or two running mod_backhand that are publicly accessible.
These front machines don't run much on them, they just proxy most requests
back to the other machines.

In the later two cases, the Backhand byAge candidacy functions will assure
that once you take down the web server on the "victim" machine, the other
servers in the cluster will not attempt to proxy requests to it.

So, as long as you can assure that your clients don't directly connect to the
machine you are taking down, then you are fine. mod_backhand will not do
anything "stupid" like try to use a machine that is marked down.

Nathan Cassano wrote:
> When you are running back-hand on a cluster and you wish to disable one of
> the machines because, for example the
> database server is down on that machine, how would you do that?
>
> Please reply to my email address as I am not subscribed to this list.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
[mod_backhand-users] (no subject) [ In reply to ]
confirm 366582
[mod_backhand-users] (no subject) [ In reply to ]
--=====================_2255906==_.ALT
Content-Type: text/plain; charset="us-ascii"; format=flowed



Hello all,

I'm new to using Backhand. Just got my first cluster of Ppros together. I
have this kind of topology

a firewall with the only public IPS 4 backhand machines that are using
NSF/NIS on a 5th superbox with 192.68.2.0/24 networking

I seem to have no problems configuring the boxes. Get the Manager up and
running, but looks like none of the other servers besides the main server
the firewall points to is getting hits.

Manager Config

<IfModule mod_backhand.c>
UnixSocketDir /var/backhand/backhand
MulticastStats 192.168.2.255:4445
AcceptStats 192.168.2.0/24
</IfModule>
<Location "/backhand/">
SetHandler backhand-handler
</Location>
<Directory />
AllowOverride All
Options None
Order allow,deny
Allow from all
Backhand byCPU
Backhand byAge
Backhand byLoad
</Directory>


Clients Configs

<IfModule mod_backhand.c>
# This directorty must be readable and writable by euid of apache (nobody)
# The mod_backhand-Arriba and the AF_UNIX domain files are stored in here.
UnixSocketDir /var/backhand/backhand
# This would be the way to do ethernet broadcast
# MulticastStats 128.220.221.255:4445
# but we choose to use the following IP multicast address with a TTL of 1
MulticastStats 192.168.2.20 192.168.2.8:4445
# We want to accept resource information originating from any IP on our
# network.
# AcceptStats 192.168.2.0/24
</IfModule>

this is the error I get when I turn on backhand with this config:

Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.
Hint: https://www.domain.com:443/


The wierd thing is the backhand manager show the address with ssl
192.168.2.20:443
192.168.8:443,192.168.2.30:443,192.168.2.40:443

what is going on here?

One last quick note I am using the Port install for Freebsd 4.3 that I
modified to work for backhand 1.2.2








Does backhand even work from behind a firewall?
--=====================_2255906==_.ALT
Content-Type: text/html; charset="us-ascii"

<html>
<x-sigsep><p></x-sigsep>
Hello all,<br>
<br>
I'm new to using Backhand. Just got my first cluster of Ppros together. I
have this kind&nbsp; of topology<br>
<br>
a firewall with the only public IPS 4 backhand machines that are using
NSF/NIS on a 5th superbox with 192.68.2.0/24 networking<br>
<br>
I seem to have no problems configuring the boxes. Get the Manager up and
running, but looks like none of the other servers besides the main server
the firewall points to is getting hits.<br>
<br>
Manager Config <br>
<br>
&lt;IfModule mod_backhand.c&gt;<br>
&nbsp;&nbsp;&nbsp; UnixSocketDir /var/backhand/backhand<br>
&nbsp;&nbsp;&nbsp; MulticastStats 192.168.2.255:4445<br>
&nbsp;&nbsp;&nbsp; AcceptStats 192.168.2.0/24<br>
&lt;/IfModule&gt;<br>
&lt;Location &quot;/backhand/&quot;&gt;<br>
&nbsp;&nbsp;&nbsp; SetHandler backhand-handler<br>
&lt;/Location&gt;<br>
&lt;Directory /&gt;<br>
&nbsp;&nbsp;&nbsp; AllowOverride All<br>
&nbsp;&nbsp;&nbsp; Options None<br>
&nbsp;&nbsp;&nbsp; Order allow,deny<br>
&nbsp;&nbsp;&nbsp; Allow from all<br>
&nbsp;&nbsp;&nbsp; Backhand byCPU<br>
&nbsp;&nbsp;&nbsp; Backhand byAge<br>
&nbsp;&nbsp;&nbsp; Backhand byLoad<br>
&lt;/Directory&gt;<br>
<br>
<br>
Clients Configs<br>
<br>
&lt;IfModule mod_backhand.c&gt;<br>
# This directorty must be readable and writable by euid of apache
(nobody)<br>
#&nbsp; The mod_backhand-Arriba and the AF_UNIX domain files are stored
in here.<br>
&nbsp;&nbsp;&nbsp; UnixSocketDir /var/backhand/backhand<br>
#&nbsp;&nbsp;&nbsp; This would be the way to do ethernet broadcast<br>
#&nbsp;&nbsp;&nbsp; MulticastStats 128.220.221.255:4445<br>
#&nbsp;&nbsp;&nbsp; but we choose to use the following IP multicast
address with a TTL of 1<br>
&nbsp;&nbsp;&nbsp; MulticastStats 192.168.2.20 192.168.2.8:4445<br>
#&nbsp;&nbsp;&nbsp; We want to accept resource information originating
from any IP on our<br>
#&nbsp;&nbsp;&nbsp;&nbsp; network.<br>
&nbsp;#&nbsp;&nbsp; AcceptStats 192.168.2.0/24<br>
&lt;/IfModule&gt;<br>
<br>
this is the error I get when I turn on backhand with this config:<br>
<br>
<font size=6><b>Bad Request<br>
</b></font>Your browser sent a request that this server could not
understand. <br>
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br>
Instead use the HTTPS scheme to access this URL, please.<br>
Hint:
<a href="https://www.domain.com/" eudora="autourl"><font color="#0000FF"><b><u>https://www.domain.com:443/<br>
<br>
</a></u></b></font><x-sigsep><p></x-sigsep>
The wierd thing is the backhand manager show the address with ssl
192.168.2.20:443<br>
192.168.8:443,192.168.2.30:443,192.168.2.40:443<br>
<br>
what is going on here?<br>
<br>
One last quick note I am using the Port install for Freebsd 4.3&nbsp;
that I modified to work for backhand 1.2.2<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<x-sigsep><p></x-sigsep>
Does backhand even work from behind a firewall?</html>

--=====================_2255906==_.ALT--
[mod_backhand-users] (no subject) [ In reply to ]
On Thursday, September 13, 2001, at 10:17 AM, Chris Manjoine wrote:
> a firewall with the only public IPS 4 backhand machines that are using=20=

> NSF/NIS on a 5th superbox with 192.68.2.0/24 networking
>
> I seem to have no problems configuring the boxes. Get the Manager up=20=

> and running, but looks like none of the other servers besides the main=20=

> server the firewall points to is getting hits.

mod_backhand uses a peer model. It expects everyone to have a similar=20=

multicast and accept configuration. If _you_ choose to put only one of=20=

them as a single point proxy, it will work fine... mod_backhand still=20
views it as a peer relationship with a grossly unbalanced input.

> Manager Config
>
> <IfModule mod_backhand.c>
> =A0=A0=A0 UnixSocketDir /var/backhand/backhand
> =A0=A0=A0 MulticastStats 192.168.2.255:4445
> =A0=A0=A0 AcceptStats 192.168.2.0/24
> </IfModule>
> <Location "/backhand/">
> =A0=A0=A0 SetHandler backhand-handler
> </Location>
> <Directory />
> =A0=A0=A0 AllowOverride All
> =A0=A0=A0 Options None
> =A0=A0=A0 Order allow,deny
> =A0=A0=A0 Allow from all
> =A0=A0=A0 Backhand byCPU
> =A0=A0=A0 Backhand byAge
> =A0=A0=A0 Backhand byLoad
> </Directory>
>
>
> Clients Configs
>
> <IfModule mod_backhand.c>
> # This directorty must be readable and writable by euid of apache=20
> (nobody)
> #=A0 The mod_backhand-Arriba and the AF_UNIX domain files are stored =
in=20
> here.
> =A0=A0=A0 UnixSocketDir /var/backhand/backhand
> #=A0=A0=A0 This would be the way to do ethernet broadcast
> #=A0=A0=A0 MulticastStats 128.220.221.255:4445
> #=A0=A0=A0 but we choose to use the following IP multicast address =
with a TTL=20
> of 1
> =A0=A0=A0 MulticastStats 192.168.2.20 192.168.2.8:4445

If these machines only have one interface, just use:
MulticastStats 192.168.2.255:4445

It will use the right IP address and the information will be available=20=

everywhere.

> #=A0=A0=A0 We want to accept resource information originating from any =
IP on=20
> our
> #=A0=A0=A0=A0 network.
> =A0#=A0=A0 AcceptStats 192.168.2.0/24

Uncomment the above line.

> </IfModule>
>

> The wierd thing is the backhand manager show the address with ssl=20
> 192.168.2.20:443
> 192.168.8:443,192.168.2.30:443,192.168.2.40:443
>
> what is going on here?

mod_backhand doesn't know which port your web server is running on. If=20=

you look in your configuration file and see:
Listen 80
Listen 443

Try reversing these, so it looks like:
Listen 443
Listen 80

Until you get your servers to show up on the /backhand/ page as IP:80,=20=

you will have a problem. All of the server to server communication=20
happen over HTTP, not HTTPS. So, the port listed on the /backhand/ page=20=

must specify a port that understands normal HTTP (non-SSL) requests.

> One last quick note I am using the Port install for Freebsd 4.3=A0 =
that I=20
> modified to work for backhand 1.2.2

1.2.0, I assume? 1.2.1 is in CVS, and 1.2.2 has not been conceived ;-)

> Does backhand even work from behind a firewall?

Should work fine.

--
Theo Schlossnagle
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
[mod_backhand-users] (no subject) [ In reply to ]
notes below your comments and suggestions

.At 10:40 AM 9/13/01 -0400, you wrote:
>On Thursday, September 13, 2001, at 10:17 AM, Chris Manjoine wrote:
>>a firewall with the only public IPS 4 backhand machines that are using
>>NSF/NIS on a 5th superbox with 192.68.2.0/24 networking
>>
>>I seem to have no problems configuring the boxes. Get the Manager up and
>>running, but looks like none of the other servers besides the main server
>>the firewall points to is getting hits.
>
>mod_backhand uses a peer model. It expects everyone to have a similar
>multicast and accept configuration. If _you_ choose to put only one of
>them as a single point proxy, it will work fine... mod_backhand still
>views it as a peer relationship with a grossly unbalanced input.
>
>>Manager Config
>>
>><IfModule mod_backhand.c>
>> UnixSocketDir /var/backhand/backhand
>> MulticastStats 192.168.2.255:4445
>> AcceptStats 192.168.2.0/24
>></IfModule>
>><Location "/backhand/">
>> SetHandler backhand-handler
>></Location>
>><Directory />
>> AllowOverride All
>> Options None
>> Order allow,deny
>> Allow from all
>> Backhand byCPU
>> Backhand byAge
>> Backhand byLoad
>></Directory>
>>
>>
>>Clients Configs
>>
>><IfModule mod_backhand.c>
>># This directorty must be readable and writable by euid of apache (nobody)
>># The mod_backhand-Arriba and the AF_UNIX domain files are stored in here.
>> UnixSocketDir /var/backhand/backhand
>># This would be the way to do ethernet broadcast
>># MulticastStats 128.220.221.255:4445
>># but we choose to use the following IP multicast address with a TTL of 1
>> MulticastStats 192.168.2.20 192.168.2.8:4445
>
when I do that I get an additional entry for the public IP interface which
is why I did it as shown above.


>If these machines only have one interface, just use:
>MulticastStats 192.168.2.255:4445
>
>It will use the right IP address and the information will be available
>everywhere.
>
>># We want to accept resource information originating from any IP on our
>># network.
>> # AcceptStats 192.168.2.0/24
>
>Uncomment the above line.
Ah So I need this event thought I am not managing via this server ok great
might want to put that in your default Comments.


>></IfModule>
>
>>The wierd thing is the backhand manager show the address with ssl
>>192.168.2.20:443
>>192.168.8:443,192.168.2.30:443,192.168.2.40:443
>>
>>what is going on here?
>
>mod_backhand doesn't know which port your web server is running on. If
>you look in your configuration file and see:
>Listen 80
>Listen 443
>
>Try reversing these, so it looks like:
>Listen 443
>Listen 80

doesn't seem to be working. I am going to try all your suggestions however
and get back with you

>Until you get your servers to show up on the /backhand/ page as IP:80, you
>will have a problem. All of the server to server communication happen
>over HTTP, not HTTPS. So, the port listed on the /backhand/ page must
>specify a port that understands normal HTTP (non-SSL) requests.
>
>>One last quick note I am using the Port install for Freebsd 4.3 that I
>>modified to work for backhand 1.2.2
>
>1.2.0, I assume? 1.2.1 is in CVS, and 1.2.2 has not been conceived ;-)
>yah 1.2.0 I am going to submit it to FreeBSD when I get it working on my boxes




>>Does backhand even work from behind a firewall?
>
>Should work fine.
>
>--
>Theo Schlossnagle
>1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>
>_______________________________________________
>backhand-users mailing list
>backhand-users@lists.backhand.org
>http://lists.backhand.org/mailman/listinfo/backhand-users
[mod_backhand-users] (no subject) [ In reply to ]
notes below your comments and suggestions

.At 10:40 AM 9/13/01 -0400, you wrote:
>On Thursday, September 13, 2001, at 10:17 AM, Chris Manjoine wrote:
>>a firewall with the only public IPS 4 backhand machines that are using
>>NSF/NIS on a 5th superbox with 192.68.2.0/24 networking
>>
>>I seem to have no problems configuring the boxes. Get the Manager up and
>>running, but looks like none of the other servers besides the main server
>>the firewall points to is getting hits.
>
>mod_backhand uses a peer model. It expects everyone to have a similar
>multicast and accept configuration. If _you_ choose to put only one of
>them as a single point proxy, it will work fine... mod_backhand still
>views it as a peer relationship with a grossly unbalanced input.
>
>>Manager Config
>>
>><IfModule mod_backhand.c>
>> UnixSocketDir /var/backhand/backhand
>> MulticastStats 192.168.2.255:4445
>> AcceptStats 192.168.2.0/24
>></IfModule>
>><Location "/backhand/">
>> SetHandler backhand-handler
>></Location>
>><Directory />
>> AllowOverride All
>> Options None
>> Order allow,deny
>> Allow from all
>> Backhand byCPU
>> Backhand byAge
>> Backhand byLoad
>></Directory>
>>
>>
>>Clients Configs
>>
>><IfModule mod_backhand.c>
>># This directorty must be readable and writable by euid of apache (nobody)
>># The mod_backhand-Arriba and the AF_UNIX domain files are stored in here.
>> UnixSocketDir /var/backhand/backhand
>># This would be the way to do ethernet broadcast
>># MulticastStats 128.220.221.255:4445
>># but we choose to use the following IP multicast address with a TTL of 1
>> MulticastStats 192.168.2.20 192.168.2.8:4445
>
when I do that I get an additional entry for the public IP interface which
is why I did it as shown above.


>If these machines only have one interface, just use:
>MulticastStats 192.168.2.255:4445
>
>It will use the right IP address and the information will be available
>everywhere.
>
>># We want to accept resource information originating from any IP on our
>># network.
>> # AcceptStats 192.168.2.0/24
>
>Uncomment the above line.
Ah So I need this event thought I am not managing via this server ok great
might want to put that in your default Comments.


>></IfModule>
>
>>The wierd thing is the backhand manager show the address with ssl
>>192.168.2.20:443
>>192.168.8:443,192.168.2.30:443,192.168.2.40:443
>>
>>what is going on here?
>
>mod_backhand doesn't know which port your web server is running on. If
>you look in your configuration file and see:
>Listen 80
>Listen 443
>
>Try reversing these, so it looks like:
>Listen 443
>Listen 80

doesn't seem to be working. I am going to try all your suggestions however
and get back with you

>Until you get your servers to show up on the /backhand/ page as IP:80, you
>will have a problem. All of the server to server communication happen
>over HTTP, not HTTPS. So, the port listed on the /backhand/ page must
>specify a port that understands normal HTTP (non-SSL) requests.
>
>>One last quick note I am using the Port install for Freebsd 4.3 that I
>>modified to work for backhand 1.2.2
>
>1.2.0, I assume? 1.2.1 is in CVS, and 1.2.2 has not been conceived ;-)
>yah 1.2.0 I am going to submit it to FreeBSD when I get it working on my boxes




>>Does backhand even work from behind a firewall?
>
>Should work fine.
>
>--
>Theo Schlossnagle
>1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>
>_______________________________________________
>backhand-users mailing list
>backhand-users@lists.backhand.org
>http://lists.backhand.org/mailman/listinfo/backhand-users