Mailing List Archive: [lvs-users] FTP persistence template synced between director nodes?

[lvs-users] FTP persistence template synced between director nodes?

Feb 1, 2019, 1:31 PM

Post #1 of 4 (496 views)

We are having some issues where when using a catch-all persistence VIP for
an FTP service and not any of the ftp ip_vs kernel modules. Sometimes when
a client sends the request to the data port it lands on the wrong backend
FTP server instead of the original control sessions backend. We have
multiple director nodes in an active/active setup using anycast/BGP to get
traffic to the nodes. So it is possible and even likely that the second
connection to the data port would land on a different director node than
the control session. We do have session syncing working between these
nodes.

Is the persistence template state synced between nodes? Or, does each node
create its own persistence template and could hash to a different backend
than the other nodes? We are using WRR right now for the lb algorithm.

We are running on Centos 7.6 kernel 3.10.0-957

Example of our configuration:

-A -t vip:0 -s wrr -p 600
-a -t vip:0 -r realip_1:0 -i -w 1
-a -t vip:0 -r realip_2:0 -i -w 1
-a -t vip:0 -r realip_3:0 -i -w 1
-a -t vip:0 -r realip_4:0 -i -w 1

Thank you,

Phillip Moore
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] FTP persistence template synced between director nodes? [ In reply to ]

ja at ssi

Feb 3, 2019, 11:30 AM

Post #2 of 4 (494 views)

Permalink

Hello,

On Fri, 1 Feb 2019, Phillip Moore wrote:

> We are having some issues where when using a catch-all persistence VIP for
> an FTP service and not any of the ftp ip_vs kernel modules. Sometimes when
> a client sends the request to the data port it lands on the wrong backend
> FTP server instead of the original control sessions backend. We have
> multiple director nodes in an active/active setup using anycast/BGP to get
> traffic to the nodes. So it is possible and even likely that the second
> connection to the data port would land on a different director node than
> the control session. We do have session syncing working between these
> nodes.
>
> Is the persistence template state synced between nodes? Or, does each node
> create its own persistence template and could hash to a different backend
> than the other nodes? We are using WRR right now for the lb algorithm.

The templates are synced but when it happens depends on the sysctl
vars. What is your configuration?

grep . /proc/sys/net/ipv4/vs/sync_*

With default settings, the templates are synced too often which
can lead to lost sync messages.

> We are running on Centos 7.6 kernel 3.10.0-957
>
> Example of our configuration:
>
> -A -t vip:0 -s wrr -p 600
> -a -t vip:0 -r realip_1:0 -i -w 1
> -a -t vip:0 -r realip_2:0 -i -w 1
> -a -t vip:0 -r realip_3:0 -i -w 1
> -a -t vip:0 -r realip_4:0 -i -w 1

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] FTP persistence template synced between director nodes? [ In reply to ]

pdm at pobox

Feb 3, 2019, 1:21 PM

Post #3 of 4 (494 views)

Permalink

On Sun, Feb 3, 2019 at 1:30 PM Julian Anastasov <ja@ssi.bg> wrote:

>
> The templates are synced but when it happens depends on the sysctl
> vars. What is your configuration?
>
> grep . /proc/sys/net/ipv4/vs/sync_*
>

We are pretty much running defaults with respect to sync I believe.

/proc/sys/net/ipv4/vs/sync_ports:1

> /proc/sys/net/ipv4/vs/sync_qlen_max:383646

> /proc/sys/net/ipv4/vs/sync_refresh_period:0

> /proc/sys/net/ipv4/vs/sync_retries:0

> /proc/sys/net/ipv4/vs/sync_sock_size:0

> /proc/sys/net/ipv4/vs/sync_threshold:3 50
/proc/sys/net/ipv4/vs/sync_version:1
>
>

We've had some unrelated issues on tcp session setup for non-ftp sessions
if the network is in such flux that the SYN and SYN-ACK land on different
hosts. This is rare and we try to avoid this happening but I think the 3
above means 3rd packet, so it wouldn't have synced with the SYN-ACK comes.

I appreciate your feedback with these settings with respect to the
persistence templates.

Thanks,
Phillip Moore
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] FTP persistence template synced between director nodes? [ In reply to ]

ja at ssi

Feb 3, 2019, 2:51 PM

Post #4 of 4 (494 views)

Permalink

Hello,

On Sun, 3 Feb 2019, Phillip Moore wrote:

> On Sun, Feb 3, 2019 at 1:30 PM Julian Anastasov <ja@ssi.bg> wrote:
>
> >
> > The templates are synced but when it happens depends on the sysctl
> > vars. What is your configuration?
> >
> > grep . /proc/sys/net/ipv4/vs/sync_*
> >
>
> We are pretty much running defaults with respect to sync I believe.
>
> /proc/sys/net/ipv4/vs/sync_ports:1
>
> > /proc/sys/net/ipv4/vs/sync_qlen_max:383646
>
> > /proc/sys/net/ipv4/vs/sync_refresh_period:0
>
> > /proc/sys/net/ipv4/vs/sync_retries:0
>
> > /proc/sys/net/ipv4/vs/sync_sock_size:0
>
> > /proc/sys/net/ipv4/vs/sync_threshold:3 50
> /proc/sys/net/ipv4/vs/sync_version:1

Yep, look like defaults.

> We've had some unrelated issues on tcp session setup for non-ftp sessions
> if the network is in such flux that the SYN and SYN-ACK land on different
> hosts. This is rare and we try to avoid this happening but I think the 3
> above means 3rd packet, so it wouldn't have synced with the SYN-ACK comes.
>
> I appreciate your feedback with these settings with respect to the
> persistence templates.

Not sure how similar is your setup but I remember for recent
thread where SH and MH algorithms can be used:

https://marc.info/?l=linux-virtual-server&m=152823122428676&w=2

In short, you can use SH scheduler (for FTP '--sched-flags sh-port'
is not suitable). The sync_persist_mode flag comes with 3.11 kernel, not
sure whether it is present in your kernel. But may be you do not need
persistence if it was used just for FTP session purposes, the SH/MH
schedulers take care to forward both FTP commands and data to same
backend server because by default they schedule only by source address,
the source port is ignored.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users