On 16.05.2013, Samuel Winchenbach wrote:
[...]
> * I have no problem when connecting from the node running ldirectord (wrr
> works fine and connecting to all the real servers works great)
>
> * If I connect from one of the other database nodes the only real that
> works correctly is the one running ldirectord. It does not work when
> connecting to the node I am connecting from, or the other non-ldirectord
> node.
>
> Here is a little paste that kind of shows what I am doing:
> http://pastie.org/private/uufpnismlibjekjozojqg
>
>
> Any idea on what might cause this? Also if you have ideas or suggestions
> on something that might be cleaner or more elegant that would be wonderful.
Accessing loadbalanced service from a realserver usually isn't that easy,
as you need to take a look into routing of your realservers :)
With direct routing, you realserver will have the VIP as a local IP
address. "ip route get $VIP" on a realserver will report your host
will route this packet via the loopback interface, also using the VIP
address as the source address: traffic won't be send to the load balancer,
it'll be handled locally.
You're using masquerading for packet-forwarding, which requires your
realservers to send all their reply packets back via the director.
In a packet transferred with masquerading, the director replaces the
destination IP address (VIP) and port by the realserver's IP address and port
(and recalculates any tcp checksums), but the client IP address still stays
the same.
A packet
Client -> VIP
is being rewritten as
Client -> Realserver
by the director.
On the way back to the client, the director also has to perform a NAT rewrite:
Realserver -> Client
becomes
VIP -> Client
Otherwise, the client will receive traffic from an unexpected IP address
with traffic belonging to a different connection. In such a scenario,
the client is expected to refuse or drop the traffic. According to your
tcpdumps, exactly that's happening.
If all your hosts do reside on the same IP network (from terms of routing:
you're using the same subnet and netmask, e.g. all hosts are running on
192.168.122.0/24) and you don't do any special ip routing foo, your return
packets won't pass the director but will be sent directly from client to
realserver.
A client connection from test2-vm would go on like this:
test2-vm TX: test2-vm -> VIP
# Director hosts VIP, sends the NAT-rewritten packet onto the network
director RX: test2-vm -> VIP
director TX: test2-vm -> test3-vm
# Realserver test3-vm picks up the traffic, sends a reply
test3-vm RX: test2-vm -> test3-vm
test3-vm TX: test3-vm -> test2-vm
# Client test2-vm picks up the packet and is confused.
test2-vm RX: test3-vm -> test2-vm
test3-vm expected a reply from VIP, but not from test2-vm.
This unexpected packet will be refused (tcp) or dropped (udp).
Or, from your tcpdumps on test1-vm:
15:50:34.636081 IP 192.168.122.165.48390 > 192.168.122.79.3306: Flags
[S], seq 2584851851, win 14600, options [mss 1460,sackOK,TS val 27933009
ecr 0,nop,wscale 7], length 0
15:50:34.636109 IP 192.168.122.165.48390 > 192.168.122.90.3306: Flags
[S], seq 2584851851, win 14600, options [mss 1460,sackOK,TS val 27933009
ecr 0,nop,wscale 7], length 0
The first line is the incoming packet, the second line the packet
leaving the director.
tcpdump from realserver test3-vm:
15:50:34.509764 IP 192.168.122.165.48390 > 192.168.122.90.3306: Flags
[S], seq 2584851851, win 14600, options [mss 1460,sackOK,TS val 27933009
ecr 0,nop,wscale 7], length 0
15:50:34.509797 IP 192.168.122.90.3306 > 192.168.122.165.48390: Flags
[S.], seq 894915954, ack 2584851852, win 14480, options [mss
1460,sackOK,TS val 27905301 ecr 27933009,nop,wscale 7], length 0
The first line is receiving the packet from the director, the second
line the reply.
>From your testdump on the client test2-vm:
15:50:34.509318 IP 192.168.122.165.48390 > 192.168.122.79.3306: Flags
[S], seq 2584851851, win 14600, options [mss 1460,sackOK,TS val 27933009
ecr 0,nop,wscale 7], length 0
15:50:34.511755 IP 192.168.122.90.3306 > 192.168.122.165.48390: Flags
[S.], seq 894915954, ack 2584851852, win 14480, options [mss
1460,sackOK,TS val 27905301 ecr 27933009,nop,wscale 7], length 0
15:50:34.511796 IP 192.168.122.165.48390 > 192.168.122.90.3306: Flags
[R], seq 2584851852, win 0, length 0
The first line is the outgoing packet (to the director), the second
line the reply from the unexpected realserver. As a result, the client
sends the third packet, a TCP-Reset (Flags: R) to the unexpected realserver.
If you'd add "-ee" to your tcpdump, you could also verify this by
matching MAC addresses.
One way to solve this is by forcing all traffic back through the director,
including any "subnet-local" traffic. On 192.168.122.90 (test3-vm), setting
up network would look like this:
$ ip route flush dev eth0
$ ip address add 192.168.122.90/32 dev eth0
$ ip route add 192.168.122.133/32 dev eth0
$ ip route add default via 192.168.122.133 dev eth0
However, in your setup, this does have some drawbacks. For example, MySQL Cluster
will prefer sending some traffic directly between its nodes, ideally
connected to the same switch without a need for another hop.
Another way might be by using advanced routing on your realservers,
using additional routing tables.
Untested, just the basic principle:
$ echo 123 lvs >> /etc/iproute2/rt_tables
$ ip route add default 192.168.122.133 table lvsnat
$ ip rule add fwmark 0x123 table lvsnat
$ iptables -I OUTPUT -j MARK --set-mark 0x123 -p tcp --sport 3306 --src 192.168.122.90
Another one: use the iptables 'mangle'-table along with arptables and rewrite
the destination mac address.
Probably the easiest one: install a simple port forwarder on the director,
forwarding traffic from the director's IP to the VIP.
Anders
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225
Amtsgericht Montabaur HRB 6484
Vorstände: Henning Ahlert, Ralph Dommermuth, Matthias Ehrlich,
Robert Hoffmann, Andreas Hofmann, Markus Huhn, Hans-Henning Kettler,
Dr. Oliver Mauss, Jan Oetjen, Martin Witt, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to
http://lists.graemef.net/mailman/listinfo/lvs-users