Mailing List Archive: [lvs-users] Session "persistence" issues

[lvs-users] Session "persistence" issues

Sep 22, 2014, 1:04 PM

Post #1 of 4 (2315 views)

Hi guys -

I'm having an issue with a 2-node setup (similar setup to what's described here) where established sessions to a particular real server don't fail over when that real server fails. That is, if a connection exists in the LVS connection state table and that real server goes down, the connections to that real server persist, rather than being cleared from the table as I would expect.

My test in a little more detail:

1) Start my service on realserver1 ONLY and open a connection (thus forcing a connection to realserver1)
2) Stop the service on realserver1 and start it on realserver2, verifying "ipvsadm -Ln" shows realserver1 down and realserver2 up
3) Establish a "new" connection to the VIP, forcing the same source port & IP with nc
4) The connection fails, trying to connect to realserver1 (verified by tcpdump)

It appears that this is because the state table still contains an entry for "SRCIP:SRCPORT VIP:DSTPORT realserver1:DSTPORT". I am new to LVS, but I assume this is not the expected behavior, because it seems it would be a fairly typical scenario if both load balancers were, for example, behind a PAT firewall.

Can anyone shed some light on this, and how I might possibly fix it? I am new to LVS so any help is appreciated!

Cheers -

elliott barrere | 206.351.3520

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] Session "persistence" issues [ In reply to ]

elliott at barrere

Sep 22, 2014, 1:08 PM

Post #2 of 4 (2275 views)

Permalink

It appears my link was stripped. The configuration is described here:

http://gcharriere.com/blog/?p=339

elliott barrere | 206.351.3520

On Sep 22, 2014, at 1:04 PM, Elliott Barrere <elliott@barrere.us> wrote:

> Hi guys -
>
> I'm having an issue with a 2-node setup (similar setup to what's described here) where established sessions to a particular real server don't fail over when that real server fails. That is, if a connection exists in the LVS connection state table and that real server goes down, the connections to that real server persist, rather than being cleared from the table as I would expect.
>
> My test in a little more detail:
>
> 1) Start my service on realserver1 ONLY and open a connection (thus forcing a connection to realserver1)
> 2) Stop the service on realserver1 and start it on realserver2, verifying "ipvsadm -Ln" shows realserver1 down and realserver2 up
> 3) Establish a "new" connection to the VIP, forcing the same source port & IP with nc
> 4) The connection fails, trying to connect to realserver1 (verified by tcpdump)
>
> It appears that this is because the state table still contains an entry for "SRCIP:SRCPORT VIP:DSTPORT realserver1:DSTPORT". I am new to LVS, but I assume this is not the expected behavior, because it seems it would be a fairly typical scenario if both load balancers were, for example, behind a PAT firewall.
>
> Can anyone shed some light on this, and how I might possibly fix it? I am new to LVS so any help is appreciated!
>
> Cheers -
>
> elliott barrere | 206.351.3520
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] Session "persistence" issues [ In reply to ]

malcolm at loadbalancer

Sep 22, 2014, 1:43 PM

Post #3 of 4 (2264 views)

Permalink

Elliot,

Their are a couple of mechanisms to handle this, the behaviour that
you probably want is:

net.ipv4.vs.expire_nodest_conn=1

expire the entry in table immediately and inform client that
connection is closed.

Also if you are using ldirectord to manage your lvs table you would
probably use: quiescent=no
i.e. on real server failure remove the entry completely from the LVS table

On 22 September 2014 21:04, Elliott Barrere <elliott@barrere.us> wrote:
> Hi guys -
>
> I'm having an issue with a 2-node setup (similar setup to what's described here) where established sessions to a particular real server don't fail over when that real server fails. That is, if a connection exists in the LVS connection state table and that real server goes down, the connections to that real server persist, rather than being cleared from the table as I would expect.
>
> My test in a little more detail:
>
> 1) Start my service on realserver1 ONLY and open a connection (thus forcing a connection to realserver1)
> 2) Stop the service on realserver1 and start it on realserver2, verifying "ipvsadm -Ln" shows realserver1 down and realserver2 up
> 3) Establish a "new" connection to the VIP, forcing the same source port & IP with nc
> 4) The connection fails, trying to connect to realserver1 (verified by tcpdump)
>
> It appears that this is because the state table still contains an entry for "SRCIP:SRCPORT VIP:DSTPORT realserver1:DSTPORT". I am new to LVS, but I assume this is not the expected behavior, because it seems it would be a fairly typical scenario if both load balancers were, for example, behind a PAT firewall.
>
> Can anyone shed some light on this, and how I might possibly fix it? I am new to LVS so any help is appreciated!
>
> Cheers -
>
> elliott barrere | 206.351.3520
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] Session "persistence" issues [ In reply to ]

elliott at barrere

Sep 22, 2014, 5:00 PM

Post #4 of 4 (2264 views)

Permalink

Thanks Malcolm, that's great. I am a little curious why that isn't the default, since it seems less common that someone would go to the trouble of managing the table through userspace, but what do I know? :)

On Sep 22, 2014, at 1:43 PM, Malcolm Turnbull <malcolm@loadbalancer.org> wrote:

> Elliot,
>
> Their are a couple of mechanisms to handle this, the behaviour that
> you probably want is:
>
> net.ipv4.vs.expire_nodest_conn=1
>
> expire the entry in table immediately and inform client that
> connection is closed.
>
>
> Also if you are using ldirectord to manage your lvs table you would
> probably use: quiescent=no
> i.e. on real server failure remove the entry completely from the LVS table
>
>
>
>
>
> On 22 September 2014 21:04, Elliott Barrere <elliott@barrere.us> wrote:
>> Hi guys -
>>
>> I'm having an issue with a 2-node setup (similar setup to what's described here) where established sessions to a particular real server don't fail over when that real server fails. That is, if a connection exists in the LVS connection state table and that real server goes down, the connections to that real server persist, rather than being cleared from the table as I would expect.
>>
>> My test in a little more detail:
>>
>> 1) Start my service on realserver1 ONLY and open a connection (thus forcing a connection to realserver1)
>> 2) Stop the service on realserver1 and start it on realserver2, verifying "ipvsadm -Ln" shows realserver1 down and realserver2 up
>> 3) Establish a "new" connection to the VIP, forcing the same source port & IP with nc
>> 4) The connection fails, trying to connect to realserver1 (verified by tcpdump)
>>
>> It appears that this is because the state table still contains an entry for "SRCIP:SRCPORT VIP:DSTPORT realserver1:DSTPORT". I am new to LVS, but I assume this is not the expected behavior, because it seems it would be a fairly typical scenario if both load balancers were, for example, behind a PAT firewall.
>>
>> Can anyone shed some light on this, and how I might possibly fix it? I am new to LVS so any help is appreciated!
>>
>> Cheers -
>>
>> elliott barrere | 206.351.3520
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users