Hi,
One of the more obvious issues with direct routing and tunneling mode in general looks like this:
-the balanced service is running at a VIP address.
-keepalived/ldirectord does perform some availability check
on the realserver's management IP address, e.g. perform some
HTTP request.
-The realserver's management IP address is different than the VIP address,
and it doesn't really tell that the VIP address is actually configured
or the service is running on the VIP address. As a result, the balancer
might be balancing load to a misconfigured realserver.
At first sight, there are the following workarounds for this:
-accept the situation as it is. E.g. binding the webserver to
0.0.0.0, connecting to one IP interface, assume it'll work anyway
on any other interface as well.
-switch to masquerading/NAT.
This requires your loadbalancer to be your gateway, and your loadbalancer
in turn would have to transport a lot more traffic than in DR/TUN mode,
which impacts scalability. It also replaces the original destination
IP address, and so a realserver probably won't know the original
destination IP address. This is an issue if a realserver is serving
multiple VIPs.
-have a port forwarder running on the realserver, forwarding traffic for
a specific port at the realserver's management IP address to the
service port at the VIP.
The balancer can connect to that port forward and check the "actual"
service by using that port forward as a proxy. Risk: if the IP address
is not configured on the realserver, the port forward may be routed
to the actual VIP back to the loadbalancer, and so probably contact
a working service at some other realserver. The check will succeed,
and so the (misconfigured) realserver put into rotation - resulting
in errors.
-check locally on every realserver if the service is configured
correctly, and whenever the balancer is doing its service checks
at the management IP address, reply with the local result.
This may be as simple as adding some check_http script to (x)inetd
and asking e.g. keepalived to check for "OK" output on that tcp port.
Same risk as in the port forwarding scenario.
-manually craft packets on the loadbalancer to check the availability
of the IP address or TCP service at the given realserver.
The later one can be implemented by combining iproute2 with
the "nping" utility from nmap:
$ nping --arp --count 1 192.0.2.10
... to get the current MAC address for realserver 192.0.2.10, e.g.
01:23:45:67:89:ab
$ ip route get 192.0.2.10
gives the source IP address used for contacting the realserver, according
to the balancer's local routing table for unspecified connections, e.g.
192.0.2.10 dev eth1 src 192.0.2.2
... so an application requesting a connection to 192.0.2.10 without a specific source IP address would result in a connection from 192.0.2.2 to 192.0.2.10.
Assuming a VIP address of 192.0.2.20, we can combine the crafted packet like this:
$ nping --dest-mac 01:23:45:67:89:ab --ttl 1 --source-ip 192.0.2.2 --dest-ip 192.0.2.20 --count 1 --interface eth1 --icmp
If the resulting nping output gives a line like this:
RCVD .* ICMP [192.0.2.20 > 192.0.2.2 Echo reply .*
... we've successfully received an icmp reply from the balanced IP address and
so know the balanced IP address is actually configured on this realserver.
One may also use "nping --tcp" to craft a TCP SYN packet for the balanced IP address and check for the Syn/Ack-packet - which essentially tells that the realserver does not only have the IP address configured, but also does perform the next step of a full 3way-handshake: there's some application trying to accept incoming connections. I'm fully aware that this is not a fully working test, but it's worth an additional check to prevent certain kinds of misconfigurations.
Another option for crafting a full test session is to create seperate network namespaces for every realserver, adding distinct routing tables within every single network namespace for the VIP to be routed via the management IP of a specific realserver, and perform every realserver's check within that specifific namespace. This involves setting up multiple veth-interfaces and bridging them with acutal interfaces, setting up multiple routing tables, ... I'm omitting an example :-)
At best, I'd like to have two checks in keepalived:
-verify every real server to respond at its VIP address (crafted icmp packet)
-check some service at the real server's management IP address to confirm
the application to be okay.
Did anyone on this list come across those issues and did probably develop a solution for this?
Best,
Anders
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225
Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
One of the more obvious issues with direct routing and tunneling mode in general looks like this:
-the balanced service is running at a VIP address.
-keepalived/ldirectord does perform some availability check
on the realserver's management IP address, e.g. perform some
HTTP request.
-The realserver's management IP address is different than the VIP address,
and it doesn't really tell that the VIP address is actually configured
or the service is running on the VIP address. As a result, the balancer
might be balancing load to a misconfigured realserver.
At first sight, there are the following workarounds for this:
-accept the situation as it is. E.g. binding the webserver to
0.0.0.0, connecting to one IP interface, assume it'll work anyway
on any other interface as well.
-switch to masquerading/NAT.
This requires your loadbalancer to be your gateway, and your loadbalancer
in turn would have to transport a lot more traffic than in DR/TUN mode,
which impacts scalability. It also replaces the original destination
IP address, and so a realserver probably won't know the original
destination IP address. This is an issue if a realserver is serving
multiple VIPs.
-have a port forwarder running on the realserver, forwarding traffic for
a specific port at the realserver's management IP address to the
service port at the VIP.
The balancer can connect to that port forward and check the "actual"
service by using that port forward as a proxy. Risk: if the IP address
is not configured on the realserver, the port forward may be routed
to the actual VIP back to the loadbalancer, and so probably contact
a working service at some other realserver. The check will succeed,
and so the (misconfigured) realserver put into rotation - resulting
in errors.
-check locally on every realserver if the service is configured
correctly, and whenever the balancer is doing its service checks
at the management IP address, reply with the local result.
This may be as simple as adding some check_http script to (x)inetd
and asking e.g. keepalived to check for "OK" output on that tcp port.
Same risk as in the port forwarding scenario.
-manually craft packets on the loadbalancer to check the availability
of the IP address or TCP service at the given realserver.
The later one can be implemented by combining iproute2 with
the "nping" utility from nmap:
$ nping --arp --count 1 192.0.2.10
... to get the current MAC address for realserver 192.0.2.10, e.g.
01:23:45:67:89:ab
$ ip route get 192.0.2.10
gives the source IP address used for contacting the realserver, according
to the balancer's local routing table for unspecified connections, e.g.
192.0.2.10 dev eth1 src 192.0.2.2
... so an application requesting a connection to 192.0.2.10 without a specific source IP address would result in a connection from 192.0.2.2 to 192.0.2.10.
Assuming a VIP address of 192.0.2.20, we can combine the crafted packet like this:
$ nping --dest-mac 01:23:45:67:89:ab --ttl 1 --source-ip 192.0.2.2 --dest-ip 192.0.2.20 --count 1 --interface eth1 --icmp
If the resulting nping output gives a line like this:
RCVD .* ICMP [192.0.2.20 > 192.0.2.2 Echo reply .*
... we've successfully received an icmp reply from the balanced IP address and
so know the balanced IP address is actually configured on this realserver.
One may also use "nping --tcp" to craft a TCP SYN packet for the balanced IP address and check for the Syn/Ack-packet - which essentially tells that the realserver does not only have the IP address configured, but also does perform the next step of a full 3way-handshake: there's some application trying to accept incoming connections. I'm fully aware that this is not a fully working test, but it's worth an additional check to prevent certain kinds of misconfigurations.
Another option for crafting a full test session is to create seperate network namespaces for every realserver, adding distinct routing tables within every single network namespace for the VIP to be routed via the management IP of a specific realserver, and perform every realserver's check within that specifific namespace. This involves setting up multiple veth-interfaces and bridging them with acutal interfaces, setting up multiple routing tables, ... I'm omitting an example :-)
At best, I'd like to have two checks in keepalived:
-verify every real server to respond at its VIP address (crafted icmp packet)
-check some service at the real server's management IP address to confirm
the application to be okay.
Did anyone on this list come across those issues and did probably develop a solution for this?
Best,
Anders
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225
Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users