Mailing List Archive

[lvs-users] Performance issues and optimization UDP LVS-NAT
Hi,

we are running lvs for couple of years and upgraded last month to 10G
infrastructure.

Now, we ran into some kind of performance issues:

Everything works fine until we reach at througput of 1600-2000Mbit/s and
193.906 pkt/s OUT 117.898 pkt/s IN

Then we run into the following situation:

CPU load average increases up to 22,
CPU Utilization increase up to 60%
Interface counter shows growing paket drops/discards

Setup: lvs-nat with WLC and session-persistence 60s, ubuntu 18.04 LTS,
HW is Dual-Socket 2x 12 core xeon GOLD 6146 @ 3,2GHz with
Hyperthreading enabled,

The loadbalancer is used to balance VPN UDP nat-t connections only (UDP
500/4500)

I found some article about performance issues with ip_conntrack.
On the system nf_conntrack is loaded and (apperntly) used by ip_vs.

/proc/sys/net/ipv4/vs/conntrack is set to 0

It looks like that some kind of table (nf?) reaches limitation.

Any suggestions to improve performance and/or fix this issue=

Thanks a lot.

regards Marco

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Performance issues and optimization UDP LVS-NAT [ In reply to ]
Hello,

On Tue, 17 Mar 2020, Marco Lorig wrote:

> Hi,
>
> we are running lvs for couple of years and upgraded last month to 10G
> infrastructure.
>
> Now, we ran into some kind of performance issues:
>
> Everything works fine until we reach at througput of 1600-2000Mbit/s and
> 193.906 pkt/s OUT 117.898 pkt/s IN
>
> Then we run into the following situation:
>
> CPU load average increases up to 22,
> CPU Utilization increase up to 60%
> Interface counter shows growing paket drops/discards
>
> Setup: lvs-nat with WLC and session-persistence 60s, ubuntu 18.04 LTS,
> HW is Dual-Socket 2x 12 core xeon GOLD 6146 @ 3,2GHz with
> Hyperthreading enabled,
>
> The loadbalancer is used to balance VPN UDP nat-t connections only (UDP
> 500/4500)
>
> I found some article about performance issues with ip_conntrack.
> On the system nf_conntrack is loaded and (apperntly) used by ip_vs.
>
> /proc/sys/net/ipv4/vs/conntrack is set to 0
>
> It looks like that some kind of table (nf?) reaches limitation.
>
> Any suggestions to improve performance and/or fix this issue=

Yes, when nf_conntrack is used it would be better to
set /proc/sys/net/ipv4/vs/conntrack to 1, as reported by different
users, for example:

https://marc.info/?t=134728825000003&r=1&w=2

In this case, you have to increase nf_conntrack_max sysctl var
to allow the desired number of conntracks to be created.

Another option is to use NOTRACK to disable nf conntracks just for
the IPVS traffic:

iptables -t raw -A PREROUTING -p tcp -d VIP --dport VPORT -j CT --notrack

For local clients use -A OUTPUT -o lo

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Performance issues and optimization UDP LVS-NAT [ In reply to ]
Hi Julian

Am 17.03.2020 um 14:23 schrieb Julian Anastasov:
>
> Yes, when nf_conntrack is used it would be better to
> set /proc/sys/net/ipv4/vs/conntrack to 1, as reported by different
> users, for example:
>
> https://marc.info/?t=134728825000003&r=1&w=2
>
> In this case, you have to increase nf_conntrack_max sysctl var
> to allow the desired number of conntracks to be created.
>
Ok, i will give it a try. nf_conntrack_max is set to 262144 (default?).
I would set it to 1024000. Do you have any recommondation for this
value? ip_vs_conn shows 18753 entries.


> Another option is to use NOTRACK to disable nf conntracks just for
> the IPVS traffic:
>
> iptables -t raw -A PREROUTING -p tcp -d VIP --dport VPORT -j CT --notrack
>
> For local clients use -A OUTPUT -o lo

As we do not use any iptables rule or connection tracking (except for
ipvs) on the loadbalancer, could it be an option for performance
optimization to disable nf_conntrack (like ip_conntrack in the past) or
is it essentially needed for proper ipvs functionality?

Just an additional info: we use Intel X710 10Gbit NICs.

Thanks.

regards Marco

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Performance issues and optimization UDP LVS-NAT [ In reply to ]
Hello,

On Tue, 17 Mar 2020, Marco Lorig wrote:

> Am 17.03.2020 um 14:23 schrieb Julian Anastasov:
> >
> > Yes, when nf_conntrack is used it would be better to
> > set /proc/sys/net/ipv4/vs/conntrack to 1, as reported by different
> > users, for example:
> >
> > https://marc.info/?t=134728825000003&r=1&w=2
> >
> > In this case, you have to increase nf_conntrack_max sysctl var
> > to allow the desired number of conntracks to be created.
> >
> Ok, i will give it a try. nf_conntrack_max is set to 262144 (default?).
> I would set it to 1024000. Do you have any recommondation for this
> value? ip_vs_conn shows 18753 entries.

Make sure nf_conntrack_count does not reach the nf_conntrack_max
value.

> > Another option is to use NOTRACK to disable nf conntracks just for
> > the IPVS traffic:
> >
> > iptables -t raw -A PREROUTING -p tcp -d VIP --dport VPORT -j CT --notrack
> >
> > For local clients use -A OUTPUT -o lo
>
> As we do not use any iptables rule or connection tracking (except for
> ipvs) on the loadbalancer, could it be an option for performance
> optimization to disable nf_conntrack (like ip_conntrack in the past) or
> is it essentially needed for proper ipvs functionality?

If you do not use iptables rules, you can disable it, it is
not needed for the IPVS traffic. Only IP_VS_NFCT, IP_VS_FTP and IP_VS_PE_SIP
depend on it. But as IP_VS_NFCT is not a separate module, may be you have
to compile IPVS without these features.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Performance issues and optimization UDP LVS-NAT [ In reply to ]
Hi,


Am 17.03.2020 um 16:02 schrieb Julian Anastasov:
>>> Yes, when nf_conntrack is used it would be better to
>>> set /proc/sys/net/ipv4/vs/conntrack to 1, as reported by different
>>> users, for example:
>>>
>>> https://marc.info/?t=134728825000003&r=1&w=2
>>>
>>> In this case, you have to increase nf_conntrack_max sysctl var
>>> to allow the desired number of conntracks to be created.
>>>
>> Ok, i will give it a try. nf_conntrack_max is set to 262144 (default?).
>> I would set it to 1024000. Do you have any recommondation for this
>> value? ip_vs_conn shows 18753 entries.
>
> Make sure nf_conntrack_count does not reach the nf_conntrack_max
> value.
>

Done. At the moment about 2400 VPN Clients are connected but
nf_conntrack_count is still 0. nf_conntrack_max is set on module load
but net.ipv4.vs.conntrack=1 is set AFTER ipvs config is loaded.

VPN is only UDP500/4500.

regards Marco




_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Performance issues and optimization UDP LVS-NAT [ In reply to ]
Hi

Am 17.03.2020 um 19:51 schrieb Marco Lorig:
>
> Done. At the moment about 2400 VPN Clients are connected but
> nf_conntrack_count is still 0. nf_conntrack_max is set on module load
> but net.ipv4.vs.conntrack=1 is set AFTER ipvs config is loaded.
>
> VPN is only UDP500/4500.

The setting seems not to have any affect (maybe only for TCP
balancing?). Load and drop rate still increases.

Bandwith is at 1,14gbps IN / 194mb/s out, ~160.000pkt/s in/ 100.000pkt/s out
/proc/net/ip_vs_conn: 10832
IPVS Connections (ipvsadm): 10832 (InActConn due to UDP)

Is there any other option to optimize performance/reduce paket drops?

regards Marco


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users