Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Linux Virtual Server>
  3. Users
[lvs-users] Connection count mismatch between client and LVS
khosrow.ebrahimpour at ssc-spc
Nov 26, 2012, 2:29 PM
Post #1 of 5 (2352 views)
Permalink
Hi list,

I am seeing some odd connection behaviour between LVS and clients. I regulary
see that the client->LVS connection count is higher from the client side than
when I look at it on the LVS.


For example, on the client side here's what I see:

client:~# netstat -antp | grep 389
tcp 197 0 10.10.5.247:32986 10.10.3.169:389 ESTABLISHED
1567/nslcd
tcp 0 0 10.10.5.247:33343 10.10.3.169:389 ESTABLISHED
1567/nslcd
tcp 0 0 10.10.5.247:33250 10.10.3.169:389 ESTABLISHED
1567/nslcd
tcp 0 0 10.10.5.247:32993 10.10.3.169:389 ESTABLISHED
1567/nslcd
tcp 0 0 10.10.5.247:60957 10.10.3.169:389 ESTABLISHED
1567/nslcd


And at the same time on the lvs I see:

lb1:~# ipvsadm -Lnc | grep 10.10.5.247
TCP 08:35 ESTABLISHED 10.10.5.247:33250 10.10.3.169:389 10.10.5.34:389
TCP 08:35 ESTABLISHED 10.10.5.247:33343 10.10.3.169:389 10.10.5.34:389
TCP 06:16 ESTABLISHED 10.10.5.247:32993 10.10.3.169:389 10.10.5.38:389


I am trying to understand why the client has 5 established connections to LVS,
while the LVS only has 3 open to the client. Is this normal behaviour?

I should add that I see this regularly on many systems.

* The clients are all running Ubuntu (10.04 or 12.04) with nslcd 0.7.2
* The LVS is running Ubuntu 12.04, Kernel 3.2.0-24-virtual, ipvsadm
1:1.25.clean-1ubuntu5

Thanks in advance,
--
Khosrow

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Connection count mismatch between client and LVS [ In reply to ]
dave at w-a-w
Nov 27, 2012, 7:34 PM
Post #2 of 5 (2272 views)
Permalink
On 11/26/2012 02:29 PM, Khosrow Ebrahimpour wrote:
> Hi list,
>
> <snip>
>
> I am trying to understand why the client has 5 established connections to LVS,
> while the LVS only has 3 open to the client. Is this normal behaviour?
>
>


Do you perhaps have a health checker of some sort running that could
account for the two connections? (Two LVS w/ keepalived, etc will look
exactly like what you have shown.)



----

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Connection count mismatch between client and LVS [ In reply to ]
khosrow.ebrahimpour at ssc-spc
Nov 28, 2012, 9:51 AM
Post #3 of 5 (2263 views)
Permalink
On November 27, 2012 07:34:20 PM David wrote:
> On 11/26/2012 02:29 PM, Khosrow Ebrahimpour wrote:
> > Hi list,
> >
> > <snip>
> >
> > I am trying to understand why the client has 5 established connections to
> > LVS, while the LVS only has 3 open to the client. Is this normal
> > behaviour?
> Do you perhaps have a health checker of some sort running that could
> account for the two connections? (Two LVS w/ keepalived, etc will look
> exactly like what you have shown.)
>
>
>
> ----

As far as I know, there's no healthcheck being done from the client to the
server. However, this may be built into the client software and I'm just not
aware of it.

To provide more context:

* the client software is nlscd (an ldap client)
* the RIP are running slapd

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Connection count mismatch between client and LVS [ In reply to ]
david at davidcoulson
Nov 28, 2012, 10:01 AM
Post #4 of 5 (2255 views)
Permalink
How long do these TCP connections live for? Perhaps they are sitting there with no packets for 15mins (or whatever the TCP timeout is on LVS in your configuration). Based on the timers of the 3 you were able to see, they stick around a while without any activity.

You could configure TCP keepalives on the server side so the connections have periodic traffic to update the LVS timers and avoid the LVS connections expiring before the connections are really closed on the server/client side.

On Nov 26, 2012, at 5:29 PM, Khosrow Ebrahimpour wrote:

> Hi list,
>
> I am seeing some odd connection behaviour between LVS and clients. I regulary
> see that the client->LVS connection count is higher from the client side than
> when I look at it on the LVS.
>
>
> For example, on the client side here's what I see:
>
> client:~# netstat -antp | grep 389
> tcp 197 0 10.10.5.247:32986 10.10.3.169:389 ESTABLISHED
> 1567/nslcd
> tcp 0 0 10.10.5.247:33343 10.10.3.169:389 ESTABLISHED
> 1567/nslcd
> tcp 0 0 10.10.5.247:33250 10.10.3.169:389 ESTABLISHED
> 1567/nslcd
> tcp 0 0 10.10.5.247:32993 10.10.3.169:389 ESTABLISHED
> 1567/nslcd
> tcp 0 0 10.10.5.247:60957 10.10.3.169:389 ESTABLISHED
> 1567/nslcd
>
>
> And at the same time on the lvs I see:
>
> lb1:~# ipvsadm -Lnc | grep 10.10.5.247
> TCP 08:35 ESTABLISHED 10.10.5.247:33250 10.10.3.169:389 10.10.5.34:389
> TCP 08:35 ESTABLISHED 10.10.5.247:33343 10.10.3.169:389 10.10.5.34:389
> TCP 06:16 ESTABLISHED 10.10.5.247:32993 10.10.3.169:389 10.10.5.38:389
>
>
> I am trying to understand why the client has 5 established connections to LVS,
> while the LVS only has 3 open to the client. Is this normal behaviour?
>
> I should add that I see this regularly on many systems.
>
> * The clients are all running Ubuntu (10.04 or 12.04) with nslcd 0.7.2
> * The LVS is running Ubuntu 12.04, Kernel 3.2.0-24-virtual, ipvsadm
> 1:1.25.clean-1ubuntu5
>
> Thanks in advance,
> --
> Khosrow
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Connection count mismatch between client and LVS [ In reply to ]
khosrow.ebrahimpour at ssc-spc
Dec 11, 2012, 6:20 AM
Post #5 of 5 (2210 views)
Permalink
Quick update: we found the issue thanks to David's hint and solved the
problem.

On November 28, 2012 01:01:34 PM David Coulson wrote:
> How long do these TCP connections live for? Perhaps they are sitting there
> with no packets for 15mins (or whatever the TCP timeout is on LVS in your
> configuration). Based on the timers of the 3 you were able to see, they
> stick around a while without any activity.
>
> You could configure TCP keepalives on the server side so the connections
> have periodic traffic to update the LVS timers and avoid the LVS
> connections expiring before the connections are really closed on the
> server/client side.

That was exactly our problem. The real servers were using the default value
for net.ipv4.tcp_keepalive_time which is 7200 seconds. And the LVS was using
the default tcp timeout value of 15 seconds (seen using: ipvsadm -L --timeout)

I think our problem was made more complex due to the fact that our setup is
using direct routing (LVS-DR), and so the connection from RIP -> CIP was
staying open despite the fact that the LVS had timedout that particular
connection.

Currently we've made the LVS TCP timeout value 7200 seconds, but I think as
per the recommendation above, we'll make the tcp_keepalive_timeout on the real
server shorter so that it will start to send keepalive packets before the LVS
timesout.


--
Khosrow

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal