Mailing List Archive

[lvs-users] Slow IPVS performance under Centos 6.5
Hi folks,

I've been successfully using IPVS under CentOS 5 for a long
time, and I've recently been using it with CentOS 6. Just this
week I noticed a performance problem with the CentOS 6 cluster, though,
and I'm wondering if anyone can offer advice on how to track it down.

In both cases, the IPVS setup directs incoming ssh connections
to one of several login nodes, something like this:

TCP my.host.edu:ssh wlc
-> host1.internal:ssh Masq 9 0 0
-> host2.internal:ssh Masq 10 0 0
-> host3.internal:ssh Masq 10 0 0

On the CentOS 6 cluster, I noticed that X applications like emacs
were taking a long time to open, and were very laggy. The CentOS 5
cluster doesn't act this way at all.

I did some experiments, and I find that if I bypass IPVS and
ssh to the CentOS 6 director machine, and then ssh from there to one
of the internal hosts, X applications are as fast as the CentOS 5
cluster through IPVS.

I also tried manually setting up a DNAT forwarding rule
on the CentOS 6 director machine, like this:

iptables -A PREROUTING -t nat -p tcp -d my.host.edu --dport 2201 -j DNAT --to
host1.internal:22
iptables -I FORWARD -p tcp -s -d host1.internal --dport 22 -j ACCEPT

If I then ssh to my.host.edu:2201 (ending up on host1.internal),
X applications are as fast as on CentOS 5 through IPVS.

Any suggestions on how I can debug this? The IPVS configurations
on the two clusters are identical. What should I be looking at?

Thanks in advance for any advice.

Bryan

--
========================================================================
Bryan Wright |"If you take cranberries and stew them like
Physics Department | applesauce, they taste much more like prunes
University of Virginia | than rhubarb does." -- Groucho
Charlottesville, VA 22901|
(434) 924-7218 | bryan@virginia.edu
========================================================================



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Slow IPVS performance under Centos 6.5 [ In reply to ]
Am Freitag, 23. Mai 2014, 10:44:10 schrieb Bryan K. Wright:
> Hi folks,
>
> I've been successfully using IPVS under CentOS 5 for a long
> time, and I've recently been using it with CentOS 6. Just this
> week I noticed a performance problem with the CentOS 6 cluster, though,
> and I'm wondering if anyone can offer advice on how to track it down.
>
> In both cases, the IPVS setup directs incoming ssh connections
> to one of several login nodes, something like this:
>
> TCP my.host.edu:ssh wlc
> -> host1.internal:ssh Masq 9 0 0
> -> host2.internal:ssh Masq 10 0 0
> -> host3.internal:ssh Masq 10 0 0
>
> On the CentOS 6 cluster, I noticed that X applications like emacs
> were taking a long time to open, and were very laggy. The CentOS 5
> cluster doesn't act this way at all.
>
> I did some experiments, and I find that if I bypass IPVS and
> ssh to the CentOS 6 director machine, and then ssh from there to one
> of the internal hosts, X applications are as fast as the CentOS 5
> cluster through IPVS.
>
> I also tried manually setting up a DNAT forwarding rule
> on the CentOS 6 director machine, like this:
>
> iptables -A PREROUTING -t nat -p tcp -d my.host.edu --dport 2201 -j DNAT
> --to host1.internal:22
> iptables -I FORWARD -p tcp -s -d host1.internal --dport 22 -j ACCEPT
>
> If I then ssh to my.host.edu:2201 (ending up on host1.internal),
> X applications are as fast as on CentOS 5 through IPVS.
>
> Any suggestions on how I can debug this? The IPVS configurations
> on the two clusters are identical. What should I be looking at?
>
> Thanks in advance for any advice.
>
> Bryan

I saw this problem, if the tcp checksum calculation was done on the interface.
Please use ethtool to switch tcp-checksum off in the IF driver.

--
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0162) 1650044
Fax: (089) 620 304 13
Re: [lvs-users] Slow IPVS performance under Centos 6.5 [ In reply to ]
Hi Michael,

misch@schwartzkopff.org said:
> I saw this problem, if the tcp checksum calculation was done on the interface.
> Please use ethtool to switch tcp-checksum off in the IF driver.


Thanks! That did the trick. It turned out that I needed
to turn off the checksum calculation on the internal interface of
my director. (ethtool --offload eth1 rx off tx off) With that
change, things are much better. I can even run glxgears!

Bryan

--
========================================================================
Bryan Wright |"If you take cranberries and stew them like
Physics Department | applesauce, they taste much more like prunes
University of Virginia | than rhubarb does." -- Groucho
Charlottesville, VA 22901|
(434) 924-7218 | bryan@virginia.edu
========================================================================



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users