Mailing List Archive

[lvs-users] [DSR] improve performance
Hello guys,

I'm looking for improvements for DSR balancing.

As for balancing I have nothing to complain about, it fulfills its role
perfectly. But I am suffering from high memory usage and at times high
usage of cpu.

I do not know if anyone has ever suffered from this, but I would like
suggestions on how to improve and get more performance.

I'm open to suggestions.

linux: CentOS release 6.9 (Final)
kernel: 4.4.128-1.el6.elrepo
piranha: 0.8.6-7.el6
ipvsadm: 1.26-4.el6


#free -m
total used free shared buffers cached
Mem: 128893 128298 595 0 133 586
-/+ buffers/cache: 127578 1315
Swap: 4095 0 4095


#lvs.cf
serial_no = 1
primary = 187.17.123.6
service = lvs
backup_active = 1
backup = 0.0.0.0
heartbeat = 1
heartbeat_port = 539
keepalive = 2
deadtime = 10
network = direct
debug_level = NONE

virtual XXX {
active = 1
address = XXX bondXX
vip_nmask = 255.255.255.0
port = 80
persistent = 60
pmask = 255.255.255.255
send_program = "/opt/lvs/http_check.sh XXX 80 %h"
expect = "0"
load_monitor = none
scheduler = wlc
protocol = tcp
timeout = 10
reentry = 5
quiesce_server = 1

server XXX {
address = XXX
active = 1
weight = 3
}
....


#iptables
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp --icmp-type any -j ACCEPT

-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
....


#sysctl
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.vs.expire_quiescent_template = 1
net.ipv4.vs.expire_nodest_conn = 1
net.nf_conntrack_max = 512000
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.ip_local_port_range = 1024 65535
net.core.somaxconn = 65535
net.ipv4.tcp_fin_timeout = 3
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.netfilter.nf_conntrack_tcp_timeout_established = 3600
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 20
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 20
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 20
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 20
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 20
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 20
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 20
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 20
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 20
net.ipv4.tcp_max_tw_buckets = 1440000
net.core.netdev_max_backlog = 200000
net.ipv4.udp_rmem_min = 16384
net.ipv4.tcp_wmem = 4096 87380 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 400000
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fastopen = 3

---

Thanks
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] [DSR] improve performance [ In reply to ]
Hello,

On Sun, 10 Jun 2018, Derick Fontes wrote:

> Hello guys,
>
> I'm looking for improvements for DSR balancing.
>
> As for balancing I have nothing to complain about, it fulfills its role
> perfectly. But I am suffering from high memory usage and at times high
> usage of cpu.
>
> I do not know if anyone has ever suffered from this, but I would like
> suggestions on how to improve and get more performance.
>
> I'm open to suggestions.

You can save memory/cycles by disabling conntrack for
the IPVS traffic. Try such rules and report if there is any gain:

# Remote clients:
iptables -t raw -A PREROUTING -p tcp -d VIP --dport VPORT -j CT --notrack

# If using local clients to VIP:VPORT:
iptables -t raw -A OUTPUT -o lo -p tcp -d VIP --dport VPORT -j CT --notrack

You will not be able to match by RELATED,ESTABLISHED but
-m state --state UNTRACKED should work, if needed at all.

> linux: CentOS release 6.9 (Final)
> kernel: 4.4.128-1.el6.elrepo
> piranha: 0.8.6-7.el6
> ipvsadm: 1.26-4.el6
>
>
> #free -m
> total used free shared buffers cached
> Mem: 128893 128298 595 0 133 586

Interesting :) Just 128MB?

> -/+ buffers/cache: 127578 1315
> Swap: 4095 0 4095

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] [DSR] improve performance [ In reply to ]
Thanks for reply,

We have a bug in the "NSS" that is used by curl in our process of checking
the state of the backend that runs by "nanny".

Basically:
https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1044666

After resolving this, there was still high CPU usage.

To better illustrate my environment, look at the number of "nanny" processes

$ ps aux | grep nanny | wc-1
913

We have basically 64 VIPs with some backends and the nanny is simply
consuming a lot of CPU.

Does anyone have this? Suggestions on how to improve this scenario?

It really is pessimistic, I have peaks of 80% -100% CPU usage.

2018-06-11 15:51 GMT-03:00 Julian Anastasov <ja@ssi.bg>:

>
> Hello,
>
> On Sun, 10 Jun 2018, Derick Fontes wrote:
>
> > Hello guys,
> >
> > I'm looking for improvements for DSR balancing.
> >
> > As for balancing I have nothing to complain about, it fulfills its role
> > perfectly. But I am suffering from high memory usage and at times high
> > usage of cpu.
> >
> > I do not know if anyone has ever suffered from this, but I would like
> > suggestions on how to improve and get more performance.
> >
> > I'm open to suggestions.
>
> You can save memory/cycles by disabling conntrack for
> the IPVS traffic. Try such rules and report if there is any gain:
>
> # Remote clients:
> iptables -t raw -A PREROUTING -p tcp -d VIP --dport VPORT -j CT --notrack
>
> # If using local clients to VIP:VPORT:
> iptables -t raw -A OUTPUT -o lo -p tcp -d VIP --dport VPORT -j CT --notrack
>
> You will not be able to match by RELATED,ESTABLISHED but
> -m state --state UNTRACKED should work, if needed at all.
>
> > linux: CentOS release 6.9 (Final)
> > kernel: 4.4.128-1.el6.elrepo
> > piranha: 0.8.6-7.el6
> > ipvsadm: 1.26-4.el6
> >
> >
> > #free -m
> > total used free shared buffers cached
> > Mem: 128893 128298 595 0 133 586
>
> Interesting :) Just 128MB?
>
> > -/+ buffers/cache: 127578 1315
> > Swap: 4095 0 4095
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>



--
Derick Fontes
dbfontes@gmail.com
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users