I just want to report on my progress to get an inexpensive HA cluster
working with mod_backhand.
I've successfully installed these packages to provide enough reliability
and scaling to suit my needs. Theo, you won't need to develop
half-proxying for mod_backhand :)
A quick question for Theo though: the patch you show on your site for
forcing keep-alive for pipelined backhand proxy requests, is it
available for apache 1.3.17? I haven't been able to apply it to my
install, and I'd really like to minimize the number of spawned
connections. I guess I can still go manually through the code...
These packages were installed, and patches applied to kernel 2.4.2:
- IPVS Netfilter module for kernel 2.4
(http://www.LinuxVirtualServer.org/software/kernel-2.4/ipvs-0.2.5.tar.gz)
- Heartbeat (http://linux-ha.org/download/heartbeat-0.4.8-1.i386.rpm)
What I did to install:
- Install the heartbeat RPM
- Install ldirectord 1.25 from the IPVS distribution (replaces
ldirectord 1.11 from heartbeat)
- Install ipvsadm from the IPVS distribution
- Install the "hidden" ARP patch to kernel 2.4 (included in IPVS
distribution)
- Install the loadable module patch for IPVS to kernel 2.4 (the static
compilation method would cause my machines to not reboot)
- Compile the IPVS modules (I put them in the /etc/ha.d tree)
- Rebuild kernel and reboot
- Configure heartbeat and ldirectord (configuration examples at
ultramonkey, see below)
This is the topology I have implemented:
http://ultramonkey.sourceforge.net/ultramonkey-1.0.2beta0/topologies/example-ha-lb-5.html
Except that I have three machines, and my director machines (2) are also
servicing requests. For it to work, one hack I had to do to hearbeat. An
aliased loopback interface is required for the machines to service
requests on the virtual ip (VIP) and return directly to the clients (in
Direct Routing mode). You don't need this with masquerade.
If a loopback interface is present on a director, heartbeat will always
assume that someone else has the VIP and not try to be master. So no one
becomes master. To correct this, I applied changes (kludges, really) to
the /etc/ha.d/resource.d/IPaddr utility (which is used to detect if we
have the VIP) to ensure that:
1) when the loopback interface is present, that it is not used in the
heartbeat decision-making process
2) That the loopback interface is brought down when we acquire the VIP
address
3) That if we relinquish the VIP, that the loopback interface is brought
back up.
(see at bottom what the patch looks like (VIP 1.1.1.1 and lo:0 as
examples)
I have configured ldirectord to distribute requests with wrr (weighted
round-robin), to optimize performance and give a measure of control over
where the requests go. I use Direct Routing (a.k.a Direct Return a.k.a
half-proxy) to allow the machines to respond directly to clients.
I leave all load decisions to mod_backhand.
Now, if a director fails, the other director takes over the VIP. If an
apache server is stopped, it is taken out of the available pool.
Vice-versa. My kind of setup :)
I've found the following documentation to be very, very useful:
http://ultramonkey.sourceforge.net/
http://www.linuxvirtualserver.org/
http://linux-ha.org/
If anyone has ideas on how to maximize performance with mod_backhand,
and minimize the number of connections, I'll be happy to see them. I'm
still topping out at about 500 requests/second, whatever the number of
nodes, and I'd like to scale!
Hope it helps someone,
Dejan
------------------------------------
IPaddr patch example for proper handling of the aliases loopback
interface (VIP 1.1.1.1 and lo:0):
MAKE SURE "HIDDEN" ATTRIBUTE IS SET FOR LOOPBACK (see hidden patch for
kernel 2.4 in IPVS distrbution)
*** IPaddr.orig Wed Mar 7 00:06:35 2001
--- /etc/ha.d/resource.d/IPaddr Wed Mar 7 00:09:17 2001
***************
*** 80,83 ****
--- 80,86 ----
$IFCONFIG $IF down
ha_log "info: IP Address $BASEIP released"
+
+ # MODIFIED BY DEJ
+ /sbin/ifconfig lo:0 1.1.1.1 broadcast 1.1.1.0 netmask
255.255.255.255
}
***************
*** 89,92 ****
--- 92,97 ----
#
find_free_interface() {
+ # MODIFIED BY DEJ
+ /sbin/ifconfig lo:0 down
ipaddr=$1;
if
***************
*** 134,139 ****
# Do we already service this IP address?
#
if
! $IFCONFIG | grep "inet addr:$1 " >/dev/null 2>&1
then
exit 0 # We already own this IP address
--- 139,145 ----
# Do we already service this IP address?
#
+ # MODIFIED BY DEJ
if
! $IFCONFIG | grep "inet addr:$1 Bcast" >/dev/null 2>&1
then
exit 0 # We already own this IP address
***************
*** 182,188 ****
ip_status() {
BASEIP=`echo $1 | sed s'%/.*%%'`
if
! $IFCONFIG | grep "inet addr:$BASEIP " >/dev/null 2>&1
then
echo "running"
--- 188,195 ----
ip_status() {
+ # MODIFIED BY DEJ
BASEIP=`echo $1 | sed s'%/.*%%'`
if
! $IFCONFIG | grep "inet addr:$BASEIP Bcast" >/dev/null 2>&1
then
echo "running"
working with mod_backhand.
I've successfully installed these packages to provide enough reliability
and scaling to suit my needs. Theo, you won't need to develop
half-proxying for mod_backhand :)
A quick question for Theo though: the patch you show on your site for
forcing keep-alive for pipelined backhand proxy requests, is it
available for apache 1.3.17? I haven't been able to apply it to my
install, and I'd really like to minimize the number of spawned
connections. I guess I can still go manually through the code...
These packages were installed, and patches applied to kernel 2.4.2:
- IPVS Netfilter module for kernel 2.4
(http://www.LinuxVirtualServer.org/software/kernel-2.4/ipvs-0.2.5.tar.gz)
- Heartbeat (http://linux-ha.org/download/heartbeat-0.4.8-1.i386.rpm)
What I did to install:
- Install the heartbeat RPM
- Install ldirectord 1.25 from the IPVS distribution (replaces
ldirectord 1.11 from heartbeat)
- Install ipvsadm from the IPVS distribution
- Install the "hidden" ARP patch to kernel 2.4 (included in IPVS
distribution)
- Install the loadable module patch for IPVS to kernel 2.4 (the static
compilation method would cause my machines to not reboot)
- Compile the IPVS modules (I put them in the /etc/ha.d tree)
- Rebuild kernel and reboot
- Configure heartbeat and ldirectord (configuration examples at
ultramonkey, see below)
This is the topology I have implemented:
http://ultramonkey.sourceforge.net/ultramonkey-1.0.2beta0/topologies/example-ha-lb-5.html
Except that I have three machines, and my director machines (2) are also
servicing requests. For it to work, one hack I had to do to hearbeat. An
aliased loopback interface is required for the machines to service
requests on the virtual ip (VIP) and return directly to the clients (in
Direct Routing mode). You don't need this with masquerade.
If a loopback interface is present on a director, heartbeat will always
assume that someone else has the VIP and not try to be master. So no one
becomes master. To correct this, I applied changes (kludges, really) to
the /etc/ha.d/resource.d/IPaddr utility (which is used to detect if we
have the VIP) to ensure that:
1) when the loopback interface is present, that it is not used in the
heartbeat decision-making process
2) That the loopback interface is brought down when we acquire the VIP
address
3) That if we relinquish the VIP, that the loopback interface is brought
back up.
(see at bottom what the patch looks like (VIP 1.1.1.1 and lo:0 as
examples)
I have configured ldirectord to distribute requests with wrr (weighted
round-robin), to optimize performance and give a measure of control over
where the requests go. I use Direct Routing (a.k.a Direct Return a.k.a
half-proxy) to allow the machines to respond directly to clients.
I leave all load decisions to mod_backhand.
Now, if a director fails, the other director takes over the VIP. If an
apache server is stopped, it is taken out of the available pool.
Vice-versa. My kind of setup :)
I've found the following documentation to be very, very useful:
http://ultramonkey.sourceforge.net/
http://www.linuxvirtualserver.org/
http://linux-ha.org/
If anyone has ideas on how to maximize performance with mod_backhand,
and minimize the number of connections, I'll be happy to see them. I'm
still topping out at about 500 requests/second, whatever the number of
nodes, and I'd like to scale!
Hope it helps someone,
Dejan
------------------------------------
IPaddr patch example for proper handling of the aliases loopback
interface (VIP 1.1.1.1 and lo:0):
MAKE SURE "HIDDEN" ATTRIBUTE IS SET FOR LOOPBACK (see hidden patch for
kernel 2.4 in IPVS distrbution)
*** IPaddr.orig Wed Mar 7 00:06:35 2001
--- /etc/ha.d/resource.d/IPaddr Wed Mar 7 00:09:17 2001
***************
*** 80,83 ****
--- 80,86 ----
$IFCONFIG $IF down
ha_log "info: IP Address $BASEIP released"
+
+ # MODIFIED BY DEJ
+ /sbin/ifconfig lo:0 1.1.1.1 broadcast 1.1.1.0 netmask
255.255.255.255
}
***************
*** 89,92 ****
--- 92,97 ----
#
find_free_interface() {
+ # MODIFIED BY DEJ
+ /sbin/ifconfig lo:0 down
ipaddr=$1;
if
***************
*** 134,139 ****
# Do we already service this IP address?
#
if
! $IFCONFIG | grep "inet addr:$1 " >/dev/null 2>&1
then
exit 0 # We already own this IP address
--- 139,145 ----
# Do we already service this IP address?
#
+ # MODIFIED BY DEJ
if
! $IFCONFIG | grep "inet addr:$1 Bcast" >/dev/null 2>&1
then
exit 0 # We already own this IP address
***************
*** 182,188 ****
ip_status() {
BASEIP=`echo $1 | sed s'%/.*%%'`
if
! $IFCONFIG | grep "inet addr:$BASEIP " >/dev/null 2>&1
then
echo "running"
--- 188,195 ----
ip_status() {
+ # MODIFIED BY DEJ
BASEIP=`echo $1 | sed s'%/.*%%'`
if
! $IFCONFIG | grep "inet addr:$BASEIP Bcast" >/dev/null 2>&1
then
echo "running"