Mailing List Archive

[lvs-users] TCP timeout and established connections in DR mode
Hi everyone,

Considering that IPVS is in DR mode with persistence disabled completely
and the client and real servers are configured to handle long-lived HTTP
connections (>15min). I understand that the default TCP timeout is 15min
but t I'm confused about the impact of this timeout on already active
established connections even when the timer value hits.

For eg, with default value 15min, will the existing connection be simply
dropped or do we keep the connection table for that 5-tuple intact?

- If the connection is simply dropped, are there any signals to look for
in terms of finding out how widespread it is?
- If we keep the connection table entry, what's the new policy on this
existing connection? (Note: persistence is disabled, as I'm aware that
there's a 60s timer which reactivates the connection template)
- If this is true, should we keep TCP timeouts on production servers
lesser than 15min to ensure we're protected in terms of some
sort of abuse?

Help is greatly appreciated.

Cheers,
Abhijeet (https://abhi.host)
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] TCP timeout and established connections in DR mode [ In reply to ]
Hi everyone,

Since I couldn't find in documentation, help is appreciated on this one :)

Thanks in advance.

On Fri, May 1, 2020 at 6:38 PM Abhijeet Rastogi <abhijeet.1989@gmail.com>
wrote:

> Hi everyone,
>
> Considering that IPVS is in DR mode with persistence disabled completely
> and the client and real servers are configured to handle long-lived HTTP
> connections (>15min). I understand that the default TCP timeout is 15min
> but t I'm confused about the impact of this timeout on already active
> established connections even when the timer value hits.
>
> For eg, with default value 15min, will the existing connection be simply
> dropped or do we keep the connection table for that 5-tuple intact?
>
> - If the connection is simply dropped, are there any signals to look
> for in terms of finding out how widespread it is?
> - If we keep the connection table entry, what's the new policy on this
> existing connection? (Note: persistence is disabled, as I'm aware that
> there's a 60s timer which reactivates the connection template)
> - If this is true, should we keep TCP timeouts on production
> servers lesser than 15min to ensure we're protected in terms of some sort
> of abuse?
>
> Help is greatly appreciated.
>
> Cheers,
> Abhijeet (https://abhi.host)
>


--
Cheers,
Abhijeet (https://abhi.host)
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] TCP timeout and established connections in DR mode [ In reply to ]
Hello,

On Fri, 1 May 2020, Abhijeet Rastogi wrote:

> Hi everyone,
>
> Considering that IPVS is in DR mode with persistence disabled completely
> and the client and real servers are configured to handle long-lived HTTP
> connections (>15min). I understand that the default TCP timeout is 15min
> but t I'm confused about the impact of this timeout on already active
> established connections even when the timer value hits.

Note that these timeouts are for inactivity, not a time from
connection creation.

> For eg, with default value 15min, will the existing connection be simply
> dropped or do we keep the connection table for that 5-tuple intact?

The IPVS structure (IPVS connection) that holds the tuple is
freed 15mins after the last packet is transferred, no packets are sent to
abort the TCP connection after this period. But following packets may get
RST due to the missing IPVS connection. ipvsadm has option to tune this
period.

> - If the connection is simply dropped, are there any signals to look for
> in terms of finding out how widespread it is?

No, we do not have stat/counter for expiration in established
state.

> - If we keep the connection table entry, what's the new policy on this
> existing connection? (Note: persistence is disabled, as I'm aware that
> there's a 60s timer which reactivates the connection template)
> - If this is true, should we keep TCP timeouts on production servers
> lesser than 15min to ensure we're protected in terms of some
> sort of abuse?

DR can be abused, while for NAT we can change the state based on
packets from real server. But it depends on the balanced application.
HTTP usually is not idle for 15mins, ssh apps can use TCP-keepalive to
protect from firewalls dropping the NAT connections. So, it is up to
you to decide what idle period you can tolerate for your application.
IPVS also has sysctl vars that can release IPVS structures on memory
pressure.

Another option is to use source-based hashing (eg. MH, SH
schedulers) and sloppy mode: echo 1 > sloppy_tcp
It allows creating IPVS connections from non-SYN packets. So, even if
IPVS connection is expired, it can be re-created to the same real
server.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] TCP timeout and established connections in DR mode [ In reply to ]
Hi Julian,

First of all, thanks a lot, really appreciate the response.

>Note that these timeouts are for inactivity, not a time from
connection creation.

This was the golden information that was missing for me. Thanks for helping
out with all the other answers, everything makes sense.

>IPVS also has sysctl vars that can release IPVS structures on memory
pressure

Are you referring to drop_entry? Doc says that it is only for SYN-RECV/SYNACK
state. What about the TCP connections that have completed the "fin
handshake"? The reason I ask is, a default timeout like 15min seems a
little too high for HTTP and I suspect that there'll be a lot of stale
entries in the connection table.

Cheers

On Tue, May 5, 2020 at 11:17 AM Julian Anastasov <ja@ssi.bg> wrote:

>
> Hello,
>
> On Fri, 1 May 2020, Abhijeet Rastogi wrote:
>
> > Hi everyone,
> >
> > Considering that IPVS is in DR mode with persistence disabled completely
> > and the client and real servers are configured to handle long-lived HTTP
> > connections (>15min). I understand that the default TCP timeout is 15min
> > but t I'm confused about the impact of this timeout on already active
> > established connections even when the timer value hits.
>
> Note that these timeouts are for inactivity, not a time from
> connection creation.
>
> > For eg, with default value 15min, will the existing connection be simply
> > dropped or do we keep the connection table for that 5-tuple intact?
>
> The IPVS structure (IPVS connection) that holds the tuple is
> freed 15mins after the last packet is transferred, no packets are sent to
> abort the TCP connection after this period. But following packets may get
> RST due to the missing IPVS connection. ipvsadm has option to tune this
> period.
>
> > - If the connection is simply dropped, are there any signals to look
> for
> > in terms of finding out how widespread it is?
>
> No, we do not have stat/counter for expiration in established
> state.
>
> > - If we keep the connection table entry, what's the new policy on this
> > existing connection? (Note: persistence is disabled, as I'm aware that
> > there's a 60s timer which reactivates the connection template)
> > - If this is true, should we keep TCP timeouts on production
> servers
> > lesser than 15min to ensure we're protected in terms of some
> > sort of abuse?
>
> DR can be abused, while for NAT we can change the state based on
> packets from real server. But it depends on the balanced application.
> HTTP usually is not idle for 15mins, ssh apps can use TCP-keepalive to
> protect from firewalls dropping the NAT connections. So, it is up to
> you to decide what idle period you can tolerate for your application.
> IPVS also has sysctl vars that can release IPVS structures on memory
> pressure.
>
> Another option is to use source-based hashing (eg. MH, SH
> schedulers) and sloppy mode: echo 1 > sloppy_tcp
> It allows creating IPVS connections from non-SYN packets. So, even if
> IPVS connection is expired, it can be re-created to the same real
> server.
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>


--
Cheers,
Abhijeet (https://abhi.host)
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] TCP timeout and established connections in DR mode [ In reply to ]
Hello,

On Tue, 5 May 2020, Abhijeet Rastogi wrote:

> >IPVS also has sysctl vars that can release IPVS structures on memory
> pressure
>
> Are you referring to drop_entry? Doc says that it is only for SYN-RECV/SYNACK
> state. What about the TCP connections that have completed the "fin
> handshake"? The reason I ask is, a default timeout like 15min seems a
> little too high for HTTP and I suspect that there'll be a lot of stale
> entries in the connection table.

May be docs are incorrect but drop_entry also works for
established state to drop connections with low traffic.

15mins should be used for applications that can go in
inactive state while applications that transfer packets constantly
can safely use lower value. If max retransmission period in TCP
is 2 minutes you can select some value that covers desired number
of retransmissions, eg. 3 maximal retransmissions => 6min. Note
that in this period there will be more retransmissions with lower
period. If only NAT mode is used, timeout such as 121 should
work as we monitor the traffic from real server and its
retransmissions.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users