Hi,
First of all, this might have nothing to do with LVS, but I'm exploring
all options. Hopefully someone here can point me in the right direction.
The setup:
- 2 directors in a pacemaker cluster with floating ip's etc.
- some realservers behind it
Half of the connections are handled by LVS, the other half is done by
Varnish (running locally on the director).
What we observer when there's a large number of connections (OpenNMS
reports over 400 requests p/sec), a client sending a SYN sometimes waits
a long time for a SYN/ACK to get send by the server. I've experienced
waiting for more than a minute for the SYN/ACK to arrive.
I see on the directory that my SYN packets do arrive. The host just
doesn't do anything with them for quite some time. Here's a small
snippet from the director:
21:25:44.557421 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135062813 ecr
0,nop,wscale 7], length 0
21:25:45.546065 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135063816 ecr
0,nop,wscale 7], length 0
21:25:47.548218 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135065820 ecr
0,nop,wscale 7], length 0
21:25:51.554730 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135069824 ecr
0,nop,wscale 7], length 0
21:25:59.570857 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135077840 ecr
0,nop,wscale 7], length 0
21:25:59.570886 IP y.y.y.y.80 > x.x.x.x.43369: Flags [S.], seq
548329830, ack 1941249137, win 5792, options [mss 1460,sackOK,TS val
2126658556 ecr 135077840,nop,wscale 7], length 0
21:25:59.592085 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 1, win
115, options [nop,nop,TS val 135077873 ecr 2126658556], length 0
21:25:59.592097 IP x.x.x.x.43369 > y.y.y.y.80: Flags [P.], seq 1:105,
ack 1, win 115, options [nop,nop,TS val 135077873 ecr 2126658556],
length 104
21:25:59.592124 IP y.y.y.y.80 > x.x.x.x.43369: Flags [.], ack 105, win
46, options [nop,nop,TS val 2126658561 ecr 135077873], length 0
21:25:59.592389 IP y.y.y.y.80 > x.x.x.x.43369: Flags [P.], seq 1:384,
ack 105, win 46, options [nop,nop,TS val 2126658562 ecr 135077873],
length 383
21:25:59.622844 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 384, win
123, options [nop,nop,TS val 135077909 ecr 2126658562], length 0
21:25:59.622857 IP x.x.x.x.43369 > y.y.y.y.80: Flags [F.], seq 105, ack
384, win 123, options [nop,nop,TS val 135077909 ecr 2126658562], length
0
21:25:59.622893 IP y.y.y.y.80 > x.x.x.x.43369: Flags [F.], seq 384, ack
106, win 46, options [nop,nop,TS val 2126658569 ecr 135077909], length 0
21:25:59.639766 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 385, win
123, options [nop,nop,TS val 135077926 ecr 2126658569], length 0
x.x.x.x = my client
y.y.y.y = IP on the director
As you see, the first SYN gets sent at 21:25:44 and only gets a SYN/ACK
reply at 21:25:59. After that, the communication is as expected.
After doing some reading I've made the following adjustments to sysctl :
net.ipv4.ip_local_port_range = 18000 65535
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
I don't think the problem is on the director's side, but I'm not sure.
The fact that i see SYN packets coming in as I send them, and the host
not responding to them, makes me doubt myself again ..
Any advice is most welcome.
Thanks,
Léon
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
First of all, this might have nothing to do with LVS, but I'm exploring
all options. Hopefully someone here can point me in the right direction.
The setup:
- 2 directors in a pacemaker cluster with floating ip's etc.
- some realservers behind it
Half of the connections are handled by LVS, the other half is done by
Varnish (running locally on the director).
What we observer when there's a large number of connections (OpenNMS
reports over 400 requests p/sec), a client sending a SYN sometimes waits
a long time for a SYN/ACK to get send by the server. I've experienced
waiting for more than a minute for the SYN/ACK to arrive.
I see on the directory that my SYN packets do arrive. The host just
doesn't do anything with them for quite some time. Here's a small
snippet from the director:
21:25:44.557421 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135062813 ecr
0,nop,wscale 7], length 0
21:25:45.546065 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135063816 ecr
0,nop,wscale 7], length 0
21:25:47.548218 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135065820 ecr
0,nop,wscale 7], length 0
21:25:51.554730 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135069824 ecr
0,nop,wscale 7], length 0
21:25:59.570857 IP x.x.x.x.43369 > y.y.y.y.80: Flags [S], seq
1941249136, win 14600, options [mss 1460,sackOK,TS val 135077840 ecr
0,nop,wscale 7], length 0
21:25:59.570886 IP y.y.y.y.80 > x.x.x.x.43369: Flags [S.], seq
548329830, ack 1941249137, win 5792, options [mss 1460,sackOK,TS val
2126658556 ecr 135077840,nop,wscale 7], length 0
21:25:59.592085 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 1, win
115, options [nop,nop,TS val 135077873 ecr 2126658556], length 0
21:25:59.592097 IP x.x.x.x.43369 > y.y.y.y.80: Flags [P.], seq 1:105,
ack 1, win 115, options [nop,nop,TS val 135077873 ecr 2126658556],
length 104
21:25:59.592124 IP y.y.y.y.80 > x.x.x.x.43369: Flags [.], ack 105, win
46, options [nop,nop,TS val 2126658561 ecr 135077873], length 0
21:25:59.592389 IP y.y.y.y.80 > x.x.x.x.43369: Flags [P.], seq 1:384,
ack 105, win 46, options [nop,nop,TS val 2126658562 ecr 135077873],
length 383
21:25:59.622844 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 384, win
123, options [nop,nop,TS val 135077909 ecr 2126658562], length 0
21:25:59.622857 IP x.x.x.x.43369 > y.y.y.y.80: Flags [F.], seq 105, ack
384, win 123, options [nop,nop,TS val 135077909 ecr 2126658562], length
0
21:25:59.622893 IP y.y.y.y.80 > x.x.x.x.43369: Flags [F.], seq 384, ack
106, win 46, options [nop,nop,TS val 2126658569 ecr 135077909], length 0
21:25:59.639766 IP x.x.x.x.43369 > y.y.y.y.80: Flags [.], ack 385, win
123, options [nop,nop,TS val 135077926 ecr 2126658569], length 0
x.x.x.x = my client
y.y.y.y = IP on the director
As you see, the first SYN gets sent at 21:25:44 and only gets a SYN/ACK
reply at 21:25:59. After that, the communication is as expected.
After doing some reading I've made the following adjustments to sysctl :
net.ipv4.ip_local_port_range = 18000 65535
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 600
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
I don't think the problem is on the director's side, but I'm not sure.
The fact that i see SYN packets coming in as I send them, and the host
not responding to them, makes me doubt myself again ..
Any advice is most welcome.
Thanks,
Léon
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users