Mailing List Archive

[lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures
I have IPVS setup with 2 VIPs talking to the same real server
configured for direct server return (ie TUN type).
One vip is port 80 http and one vip is 443 for https/SSL. The SSL vip
doesn't work properly. There is initial communication that happens but
then it appears as though IPVS stops tunneling the incoming packets to
the real server and the connection stalls and times out. If I switch
ports to just verify there is nothing crazy going on with filtering
and I put SSL on port 80 (or any port) it still fails.

I've put the relevant info in a gist in hope it might be helpful and
not clutter up the email.

https://gist.github.com/realpdm/2118bbaa298ff3debe52

In various test scenarios we found that the client is having to
retransmit packets after some initial successful back and forth. On
the IPVS node a tcpdump shows that for some reason IPVS stops
forwarding the packets onto the real server over the tunnel. You can
see in the tcpdump IPVS is forwarding things over ipip just fine until
it stops around line 15 in the dump

http traffic doesn't do this at all only SSL.

I'm really puzzled and hope i am missing something obvious. I
appreciate any insights or suggestions.

OS Info:

Linux adc-ipvs-lb2001 2.6.32-504.30.3.el6.x86_64 #1 SMP Tue Jul 14
11:18:03 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux

/sbin/modinfo ip_vs
filename:
/lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko
srcversion: 6C3CC9C055045FA0ECA1774
depends: ipv6,libcrc32c
vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
parm: conn_tab_bits:Set connections' hash size (int)

/sbin/modinfo ip_vs_sh
filename:
/lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs_sh.ko
srcversion: 2EAF6C9DD83264246DBA82C
depends: ip_vs
vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions


ipvsadm-1.26-4.el6.x86_64


Thank you,
Phillip Moore

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
Hello,

On Thu, 27 Aug 2015, Phillip Moore wrote:

> I have IPVS setup with 2 VIPs talking to the same real server
> configured for direct server return (ie TUN type).
> One vip is port 80 http and one vip is 443 for https/SSL. The SSL vip
> doesn't work properly. There is initial communication that happens but
> then it appears as though IPVS stops tunneling the incoming packets to
> the real server and the connection stalls and times out. If I switch
> ports to just verify there is nothing crazy going on with filtering
> and I put SSL on port 80 (or any port) it still fails.
>
> I've put the relevant info in a gist in hope it might be helpful and
> not clutter up the email.
>
> https://gist.github.com/realpdm/2118bbaa298ff3debe52
>
> In various test scenarios we found that the client is having to
> retransmit packets after some initial successful back and forth. On
> the IPVS node a tcpdump shows that for some reason IPVS stops
> forwarding the packets onto the real server over the tunnel. You can
> see in the tcpdump IPVS is forwarding things over ipip just fine until
> it stops around line 15 in the dump
>
> http traffic doesn't do this at all only SSL.
>
> I'm really puzzled and hope i am missing something obvious. I
> appreciate any insights or suggestions.

I'm not sure what kind of fixes contain your kernel
but the problem should be related to PMTU (ICMP FRAG_NEEDED)
or GRO. In your packet trace I don't see ICMP messages,
may be there are such packets from real server to client.
For example:

17:09:40.820663 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
17:09:40.820678 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0 (ipip-proto-4)
17:09:40.820704 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 3580, win 160, options [nop,nop,TS val 592982574 ecr 2501301618], length 0

See how client acks 1449 and then 3580, i.e. 2131 bytes
were sent to client.

SSL usually sends large certificates at start, so
that may explain the difference with plain HTTP. But you will
need more traces, mostly from the real server or better its
uplink router, if possible. For TUN mode the following traces
catch all traffic:

director# tcpdump -ln -i INDEV host CIP
director# tcpdump -ln -i OUTDEV host RIP -vvv
real server# tcpdump -ln -i IN_ETH host DIP -vvv
real server# tcpdump -ln -i tunl0 host CIP
real server# tcpdump -ln -i OUT_DEV host CIP
client# tcpdump -ln host VIP

You need to catch TCP, IPIP, ICMP. Restricting
by client IP may help.

In linux-stable I see that only 2.6.32.61+ contains
related changes:

9df2c9a ipvs: IPv6 MTU checking cleanup and bugfix
ec3dc8c ipvs: allow transmit of GRO aggregated skbs

But such fixes are for director and I don't see
any ICMP messages... And if there are ICMP messages, we
have more fixes to consider...

> OS Info:
>
> Linux adc-ipvs-lb2001 2.6.32-504.30.3.el6.x86_64 #1 SMP Tue Jul 14
> 11:18:03 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> /sbin/modinfo ip_vs
> filename:
> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko
> srcversion: 6C3CC9C055045FA0ECA1774
> depends: ipv6,libcrc32c
> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
> parm: conn_tab_bits:Set connections' hash size (int)
>
> /sbin/modinfo ip_vs_sh
> filename:
> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs_sh.ko
> srcversion: 2EAF6C9DD83264246DBA82C
> depends: ip_vs
> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
I do not understand why on line 15 of the tcpdump you can see a 326
byte packet is received from the client, but isn't forwarded to the
real server. There wouldn't be any fragmentation issues with that
would there? On line 15 you can see it keeps receiving the same packet
6 times and fails to forward it on.

We do not see any ICMP on the hosts.

Thank you I appreciate the insights.



On Fri, Aug 28, 2015 at 2:42 AM, Julian Anastasov <ja@ssi.bg> wrote:
>
> Hello,
>
> On Thu, 27 Aug 2015, Phillip Moore wrote:
>
>> I have IPVS setup with 2 VIPs talking to the same real server
>> configured for direct server return (ie TUN type).
>> One vip is port 80 http and one vip is 443 for https/SSL. The SSL vip
>> doesn't work properly. There is initial communication that happens but
>> then it appears as though IPVS stops tunneling the incoming packets to
>> the real server and the connection stalls and times out. If I switch
>> ports to just verify there is nothing crazy going on with filtering
>> and I put SSL on port 80 (or any port) it still fails.
>>
>> I've put the relevant info in a gist in hope it might be helpful and
>> not clutter up the email.
>>
>> https://gist.github.com/realpdm/2118bbaa298ff3debe52
>>
>> In various test scenarios we found that the client is having to
>> retransmit packets after some initial successful back and forth. On
>> the IPVS node a tcpdump shows that for some reason IPVS stops
>> forwarding the packets onto the real server over the tunnel. You can
>> see in the tcpdump IPVS is forwarding things over ipip just fine until
>> it stops around line 15 in the dump
>>
>> http traffic doesn't do this at all only SSL.
>>
>> I'm really puzzled and hope i am missing something obvious. I
>> appreciate any insights or suggestions.
>
> I'm not sure what kind of fixes contain your kernel
> but the problem should be related to PMTU (ICMP FRAG_NEEDED)
> or GRO. In your packet trace I don't see ICMP messages,
> may be there are such packets from real server to client.
> For example:
>
> 17:09:40.820663 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
> 17:09:40.820678 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0 (ipip-proto-4)
> 17:09:40.820704 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 3580, win 160, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
>
> See how client acks 1449 and then 3580, i.e. 2131 bytes
> were sent to client.
>
> SSL usually sends large certificates at start, so
> that may explain the difference with plain HTTP. But you will
> need more traces, mostly from the real server or better its
> uplink router, if possible. For TUN mode the following traces
> catch all traffic:
>
> director# tcpdump -ln -i INDEV host CIP
> director# tcpdump -ln -i OUTDEV host RIP -vvv
> real server# tcpdump -ln -i IN_ETH host DIP -vvv
> real server# tcpdump -ln -i tunl0 host CIP
> real server# tcpdump -ln -i OUT_DEV host CIP
> client# tcpdump -ln host VIP
>
> You need to catch TCP, IPIP, ICMP. Restricting
> by client IP may help.
>
> In linux-stable I see that only 2.6.32.61+ contains
> related changes:
>
> 9df2c9a ipvs: IPv6 MTU checking cleanup and bugfix
> ec3dc8c ipvs: allow transmit of GRO aggregated skbs
>
> But such fixes are for director and I don't see
> any ICMP messages... And if there are ICMP messages, we
> have more fixes to consider...
>
>> OS Info:
>>
>> Linux adc-ipvs-lb2001 2.6.32-504.30.3.el6.x86_64 #1 SMP Tue Jul 14
>> 11:18:03 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>> /sbin/modinfo ip_vs
>> filename:
>> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko
>> srcversion: 6C3CC9C055045FA0ECA1774
>> depends: ipv6,libcrc32c
>> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
>> parm: conn_tab_bits:Set connections' hash size (int)
>>
>> /sbin/modinfo ip_vs_sh
>> filename:
>> /lib/modules/2.6.32-504.30.3.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs_sh.ko
>> srcversion: 2EAF6C9DD83264246DBA82C
>> depends: ip_vs
>> vermagic: 2.6.32-504.30.3.el6.x86_64 SMP mod_unload modversions
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
Hello,

On Fri, 28 Aug 2015, Phillip Moore wrote:

> I do not understand why on line 15 of the tcpdump you can see a 326
> byte packet is received from the client, but isn't forwarded to the
> real server. There wouldn't be any fragmentation issues with that
> would there? On line 15 you can see it keeps receiving the same packet
> 6 times and fails to forward it on.

Can you test with enabled nf_conntrack_tcp_be_liberal
or ip_conntrack_tcp_be_liberal sysctl value in director?
May be packets are dropped by conntrack because packets
from reply direction are not seen.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
Hello,

I'm the network guy working through this issue with Phillip.

Phillip Moore wrote:
>
> We do not see any ICMP on the hosts.
> On Fri, Aug 28, 2015 at 2:42 AM, Julian Anastasov <ja@ssi.bg> wrote:
> >
> > On Thu, 27 Aug 2015, Phillip Moore wrote:
> >
> >> I have IPVS setup with 2 VIPs talking to the same real server
> >> configured for direct server return (ie TUN type).
> >> One vip is port 80 http and one vip is 443 for https/SSL. The SSL vip
> >> doesn't work properly. There is initial communication that happens but
> >> then it appears as though IPVS stops tunneling the incoming packets to
> >> the real server and the connection stalls and times out. If I switch
> >> ports to just verify there is nothing crazy going on with filtering
> >> and I put SSL on port 80 (or any port) it still fails.
> >>
> >> I've put the relevant info in a gist in hope it might be helpful and
> >> not clutter up the email.
> >>
> >> https://gist.github.com/realpdm/2118bbaa298ff3debe52

I'm going to have to clutter up the email to annotate the packet
captures, so please bear with me...


> >> In various test scenarios we found that the client is having to
> >> retransmit packets after some initial successful back and forth. On
> >> the IPVS node a tcpdump shows that for some reason IPVS stops
> >> forwarding the packets onto the real server over the tunnel. You can
> >> see in the tcpdump IPVS is forwarding things over ipip just fine until
> >> it stops around line 15 in the dump
> >>
> >> http traffic doesn't do this at all only SSL.
> >>
> >> I'm really puzzled and hope i am missing something obvious. I
> >> appreciate any insights or suggestions.
> >
> > I'm not sure what kind of fixes contain your kernel
> > but the problem should be related to PMTU (ICMP FRAG_NEEDED)
> > or GRO. In your packet trace I don't see ICMP messages,
> > may be there are such packets from real server to client.
> > For example:
> >
> > 17:09:40.820663 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
> > 17:09:40.820678 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS val 592982574 ecr 2501301618], length 0 (ipip-proto-4)
> > 17:09:40.820704 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack 3580, win 160, options [nop,nop,TS val 592982574 ecr 2501301618], length 0
> >
> > See how client acks 1449 and then 3580, i.e. 2131 bytes
> > were sent to client.
>
> I do not understand why on line 15 of the tcpdump you can see a 326
> byte packet is received from the client, but isn't forwarded to the
> real server. There wouldn't be any fragmentation issues with that
> would there? On line 15 you can see it keeps receiving the same packet
> 6 times and fails to forward it on.

I want to echo Phillip that we're not seeing anything that implies MTU
problems. The director is using the same interface for ingress and
egress traffic. The trace that we took from the director seems to be a
smoking gun. I realize this will only show traffic from the client to
the director and from the director to the real server, but it clearly
shows a problem as Phillip points out regarding the 326-byte segment.

| 17:09:40.750074 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [S], seq
| 573101480, win 14600, options [mss 1460,sackOK,TS val 592982504 ecr
| 0,nop,wscale 7], length 0
|
| 17:09:40.750111 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
| 10.64.96.10.443: Flags [S], seq 573101480, win 14600, options [mss
| 1460,sackOK,TS val 592982504 ecr 0,nop,wscale 7], length 0
| (ipip-proto-4)
|
| 17:09:40.750780 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack
| 4261714231, win 115, options [nop,nop,TS val 592982504 ecr
| 2501301549], length 0
|
| 17:09:40.750796 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
| 10.64.96.10.443: Flags [.], ack 1, win 115, options [nop,nop,TS val
| 592982504 ecr 2501301549], length 0 (ipip-proto-4)

Initial 3-way handshake is completed. All packets coming into the
director from the client are forwarded to the real server in an ipip
tunnel.

| 17:09:40.819644 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [P.], seq
| 0:77, ack 1, win 115, options [nop,nop,TS val 592982573 ecr
| 2501301549], length 77
|
| 17:09:40.819659 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
| 10.64.96.10.443: Flags [P.], seq 0:77, ack 1, win 115, options
| [nop,nop,TS val 592982573 ecr 2501301549], length 77 (ipip-proto-4)

A 77-byte packet is send from client to director and forwarded as
expected.

| 17:09:40.820663 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack
| 1449, win 137, options [nop,nop,TS val 592982574 ecr 2501301618],
| length 0
|
| 17:09:40.820678 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
| 10.64.96.10.443: Flags [.], ack 1449, win 137, options [nop,nop,TS
| val 592982574 ecr 2501301618], length 0 (ipip-proto-4)

The client acknowledges 1448 bytes of data received directly from the
real server.

| 17:09:40.820704 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [.], ack
| 3580, win 160, options [nop,nop,TS val 592982574 ecr 2501301618],
| length 0

| 17:09:40.820705 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
| 10.64.96.10.443: Flags [.], ack 3580, win 160, options [nop,nop,TS
| val 592982574 ecr 2501301618], length 0 (ipip-proto-4)

The client acknowledges an additional 2132 bytes of data received
directly from the real server.

| 17:09:40.823431 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [P.], seq
| 77:403, ack 3580, win 160, options [nop,nop,TS val 592982577 ecr
| 2501301618], length 326

This is the 326-byte segment that the director receives from the client
a total of 6 times. There is no corresponding segment outbound from the
director to the real server. The director is eating this segment.

During the time that this segment is being transmitted and
retransmitted, `sudo ipvsadm -Ln -c` shows an ESTABLISHED connection.

When the client ultimately gives up (or I ^C), this packet is sent from
the client to the director:

| 17:09:46.017707 IP 10.240.8.72.60642 > 10.64.96.10.443: Flags [F.], seq
| 403, ack 3580, win 160, options [nop,nop,TS val 592987771 ecr
| 2501301618], length 0

And the director dutifully forwards the FIN to the real server:

| 17:09:46.017723 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.60642 >
| 10.64.96.10.443: Flags [F.], seq 403, ack 3580, win 160, options
| [nop,nop,TS val 592987771 ecr 2501301618], length 0 (ipip-proto-4)

So the connection still exists and is known. But the client's data
packets in the middle get lost.

> > SSL usually sends large certificates at start, so
> > that may explain the difference with plain HTTP. But you will
> > need more traces, mostly from the real server or better its
> > uplink router, if possible. For TUN mode the following traces
> > catch all traffic:

Packet size definitely has something to do with the issue. But the magic
number isn't what you'd think.

I just ran a series of tests noting that if I sent some extra headers
over plain-old-HTTP, I could also repro the issue.

This works:

% curl -kvH 'Host: test' -H 'X: AAAAAAAA' http://10.64.96.10/iptest.php

| 12:43:52.390132 IP 10.240.8.72.39812 > 10.64.96.10.http: Flags [P.], seq
| 0:190, ack 1, win 115, options [nop,nop,TS val 670634144 ecr
| 10917205], length 190
|
| 12:43:52.390135 IP 10.65.74.77 > 10.65.74.72: IP 10.240.8.72.39812 >
| 10.64.96.10.http: Flags [P.], seq 0:190, ack 1, win 115, options
| [nop,nop,TS val 670634144 ecr 10917205], length 190 (ipip-proto-4)

This fails:

% curl -kvH 'Host: test' -H 'X: AAAAAAAAA' http://10.64.96.10/iptest.php

| 12:43:55.955943 IP 10.240.8.72.39815 > 10.64.96.10.http: Flags [P.], seq
| 0:191, ack 1, win 115, options [nop,nop,TS val 670637710 ecr
| 10920771], length 191
|
| 12:43:56.156790 IP 10.240.8.72.39815 > 10.64.96.10.http: Flags [P.], seq
| 0:191, ack 1, win 115, options [nop,nop,TS val 670637911 ecr
| 10920771], length 191
|
| 12:43:56.558762 IP 10.240.8.72.39815 > 10.64.96.10.http: Flags [P.], seq
| 0:191, ack 1, win 115, options [nop,nop,TS val 670638313 ecr
| 10920771], length 191

With no corresponding packet being sent out by the director.

So TCP payload >= 190 bytes is a smashing success. 191 bytes, and the
director eats the packet.

For completeness:

| client % ping -s 1460 -c 1 $VIP
| PING 10.64.96.10 (10.64.96.10) 1460(1488) bytes of data.
| 1468 bytes from 10.64.96.10: icmp_seq=1 ttl=59 time=0.631 ms
|
| --- 10.64.96.10 ping statistics ---
| 1 packets transmitted, 1 received, 0% packet loss, time 0ms
| rtt min/avg/max/mdev = 0.631/0.631/0.631/0.000 ms
|
| director % ping -s 1460 -c 1 $REALIP
| PING 10.65.74.72 (10.65.74.72) 1460(1488) bytes of data.
| 1468 bytes from 10.65.74.72: icmp_seq=1 ttl=64 time=0.154 ms
|
| --- 10.65.74.72 ping statistics ---
| 1 packets transmitted, 1 received, 0% packet loss, time 0ms
| rtt min/avg/max/mdev = 0.154/0.154/0.154/0.000 ms

MTU is well over 1000 all the way through -- we should not be having
trouble with 191-byte segments.

Now, I'm guessing that if this were widespread behavior, nobody would be
using LVS. Which leads me to believe there must either be something
about our specific configuration or kernel version. Has nobody
encountered this type of behavior before?

> >
> > director# tcpdump -ln -i INDEV host CIP
> > director# tcpdump -ln -i OUTDEV host RIP -vvv
> > real server# tcpdump -ln -i IN_ETH host DIP -vvv
> > real server# tcpdump -ln -i tunl0 host CIP
> > real server# tcpdump -ln -i OUT_DEV host CIP
> > client# tcpdump -ln host VIP
> >
> > You need to catch TCP, IPIP, ICMP. Restricting
> > by client IP may help.

We have these captures from every point of view. Every packet tunneled
by the director is successfully received by the real server. Every
packet sent by the real server is successfully received by the client.
Every packet sent by the client is successfully received by the
director.

If you have any other ideas about specific version or debugging we could
enable, we'd really appreciate it.

--
Chris Cowart
http://www.timesinks.net/
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
Thank you for the suggestion.

We didn't have the netfilter module loaded at all so I don't think it
would have having any impact. However I loaded it and set this setting
and it didn't change the behavior.
The ip_conntrack_tcp_be_liberal setting wasn't available on our kernel
version looks like I can't find a module to load to enable that.

We did find something interesting. If we add additional headers to the
working http request we can make it fail.

WORKS: curl -H "X:1" http://10.64.96.10/healthcheck
FAILS: curl -H "X:12" http://10.64.96.10/healthcheck

190 bytes works, 191 bytes fails with the failure to tunnel problem.


Phillip Moore


On Fri, Aug 28, 2015 at 2:42 PM, Julian Anastasov <ja@ssi.bg> wrote:

>
> Can you test with enabled nf_conntrack_tcp_be_liberal
> or ip_conntrack_tcp_be_liberal sysctl value in director?
> May be packets are dropped by conntrack because packets
> from reply direction are not seen.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
Hello,

On Fri, 28 Aug 2015, Phillip Moore wrote:

> Thank you for the suggestion.
>
> We didn't have the netfilter module loaded at all so I don't think it
> would have having any impact. However I loaded it and set this setting
> and it didn't change the behavior.
> The ip_conntrack_tcp_be_liberal setting wasn't available on our kernel
> version looks like I can't find a module to load to enable that.
>
> We did find something interesting. If we add additional headers to the
> working http request we can make it fail.
>
> WORKS: curl -H "X:1" http://10.64.96.10/healthcheck
> FAILS: curl -H "X:12" http://10.64.96.10/healthcheck
>
> 190 bytes works, 191 bytes fails with the failure to tunnel problem.

What is the model/revision of the incoming network device?
Do you have IPVS debugging enabled to check if IPVS works
with these dropped packets? Or they are lost before reaching
IPVS at LOCAL_IN?

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
Well now for the facepalm moment. I updated our kernel for Scientific
Linux for a security update and it seems to have solved this problem.
I will endeavor to find what bug was fixed now.

failing kernel: kernel-2.6.32-504.30.3.el6.x86_64
working kernel: kernel-2.6.32-573.el6.x86_64

Julian, thank you so much for your help. It is really appreciated.

Phillip Moore

On Fri, Aug 28, 2015 at 3:16 PM, Phillip Moore <pdm@pobox.com> wrote:
> Thank you for the suggestion.
>
> We didn't have the netfilter module loaded at all so I don't think it
> would have having any impact. However I loaded it and set this setting
> and it didn't change the behavior.
> The ip_conntrack_tcp_be_liberal setting wasn't available on our kernel
> version looks like I can't find a module to load to enable that.
>
> We did find something interesting. If we add additional headers to the
> working http request we can make it fail.
>
> WORKS: curl -H "X:1" http://10.64.96.10/healthcheck
> FAILS: curl -H "X:12" http://10.64.96.10/healthcheck
>
> 190 bytes works, 191 bytes fails with the failure to tunnel problem.
>
>
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS stops tunneling with ipip on SSL traffic causing session failures [ In reply to ]
Do you use IPIP tunnel on the real servers ?


On Fri, Aug 28, 2015 at 2:15 PM, Julian Anastasov <ja@ssi.bg> wrote:

>
> Hello,
>
> On Fri, 28 Aug 2015, Phillip Moore wrote:
>
> > Thank you for the suggestion.
> >
> > We didn't have the netfilter module loaded at all so I don't think it
> > would have having any impact. However I loaded it and set this setting
> > and it didn't change the behavior.
> > The ip_conntrack_tcp_be_liberal setting wasn't available on our kernel
> > version looks like I can't find a module to load to enable that.
> >
> > We did find something interesting. If we add additional headers to the
> > working http request we can make it fail.
> >
> > WORKS: curl -H "X:1" http://10.64.96.10/healthcheck
> > FAILS: curl -H "X:12" http://10.64.96.10/healthcheck
> >
> > 190 bytes works, 191 bytes fails with the failure to tunnel problem.
>
> What is the model/revision of the incoming network device?
> Do you have IPVS debugging enabled to check if IPVS works
> with these dropped packets? Or they are lost before reaching
> IPVS at LOCAL_IN?
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users