Mailing List Archive

On 5/23/13 11:04 AM, daryl herzmann wrote:
> Greetings,
>
> I have a LVS-NAT web cluster that seems to be nicely chugging along. My
> LVS director has the following specs:
>
> - Dell OptiPlex 745
> - Intel(R) Pentium(R) D CPU 3.40GHz
> - 4 GB Memory
> - BCM5754 Gigabit Ethernet PCI Express (two ports) [gro is disabled!]
I was told by RedHat the GRO issue is fixed in RHEL6.4 - Not had chance
to test it yet.

David

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Hello Daryl,

Simply add second LVS cluster , split the traffik via dns RR .

Hope this helps.

--
Mit freundlichen Grüßen / Best Regards
Horst Venzke ; PGP NET : 1024G/082F2E6D ; http://www.remsnet.de
Legal Notice: This transmittal and/or attachments may be privileged or
confidential. It is intended solely for the addressee named above. Any
review, dissemination, or copying is strictly prohibited. If you received
this transmittal in error, please notify us immediately by reply and
immediately delete this message and all its attachments. Thank you.


Gesendet: Donnerstag, 23. Mai 2013 um 17:04 Uhr
Von: "daryl herzmann" <akrherz@iastate.edu>
An: lvs-users@linuxvirtualserver.org
Betreff: [lvs-users] Reasonable(?) Performance of LVS-NAT
Greetings,
I have a LVS-NAT web cluster that seems to be nicely chugging along. My
LVS director has the following specs:
- Dell OptiPlex 745
- Intel(R) Pentium(R) D CPU 3.40GHz
- 4 GB Memory
- BCM5754 Gigabit Ethernet PCI Express (two ports) [gro is disabled!]
- RHEL6.4 64bit 2.6.32-358.6.2.el6.x86_64
- ipvsadm 1.25-10
The WAN and LAN connections are plugged into that Broadcom. I have GigE
network on both sides.
My monitoring tools indicate that:
- My apache cluster maxes out around 8,000 requests per second
- The individual apache nodes do not appear to be maxing CPU or
out of available apache slots.
- The throughput on the LVS director is maxing at about 32 MB/s
- sar reporting on the LVS director shows %system maxing out at
50% , which makes me to think the director is the bottleneck?
- sar reports %user percentage is less than 1% and %idle at ~50%
- `ipvsadm -L` indicates around 5,000 active and 700,000 inactive
connections at the time of peak throughput
Users are not reporting connection issues, just general slowness. So my
questions for your kind consideration are:
1) Do I have something obviously setup wrong or are these numbers about
what you would expect for cheap hardware and a blissfully ignorant
sysadmin?
2) Am I right to think the director is the bottleneck or is there a
better way to prove that?
3) Should I dump NAT and move to DR or something else?
thanks,
daryl
_______________________________________________
Please read the documentation before posting - it's available at:
[1]http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to [2]http://lists.graemef.net/mailman/listinfo/lvs-users

References

1. http://www.linuxvirtualserver.org/
2. http://lists.graemef.net/mailman/listinfo/lvs-users
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On Thu, 23 May 2013, David Coulson wrote:

>
> On 5/23/13 11:04 AM, daryl herzmann wrote:
>> Greetings,
>>
>> I have a LVS-NAT web cluster that seems to be nicely chugging along. My
>> LVS director has the following specs:
>>
>> - Dell OptiPlex 745
>> - Intel(R) Pentium(R) D CPU 3.40GHz
>> - 4 GB Memory
>> - BCM5754 Gigabit Ethernet PCI Express (two ports) [gro is disabled!]
> I was told by RedHat the GRO issue is fixed in RHEL6.4 - Not had chance to
> test it yet.

Thanks, it appears this is the relevant bug?

https://bugzilla.redhat.com/show_bug.cgi?id=854066

I did a test of this:

# ethtool -k eth1
Features for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

and the problem did not appear to go away. The traffic through the
director was extremely sluggish, so I failed back to having 'gro off',
shrug...

daryl

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

I seem to recall a nasty bug in the older broadcom driver that ignores
the gro=off setting!
I believe the latest driver should fix it.


On 23 May 2013 17:12, daryl herzmann <akrherz@iastate.edu> wrote:
> On Thu, 23 May 2013, David Coulson wrote:
>
>>
>> On 5/23/13 11:04 AM, daryl herzmann wrote:
>>> Greetings,
>>>
>>> I have a LVS-NAT web cluster that seems to be nicely chugging along. My
>>> LVS director has the following specs:
>>>
>>> - Dell OptiPlex 745
>>> - Intel(R) Pentium(R) D CPU 3.40GHz
>>> - 4 GB Memory
>>> - BCM5754 Gigabit Ethernet PCI Express (two ports) [gro is disabled!]
>> I was told by RedHat the GRO issue is fixed in RHEL6.4 - Not had chance to
>> test it yet.
>
> Thanks, it appears this is the relevant bug?
>
> https://bugzilla.redhat.com/show_bug.cgi?id=854066
>
> I did a test of this:
>
> # ethtool -k eth1
> Features for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: off
>
> and the problem did not appear to go away. The traffic through the
> director was extremely sluggish, so I failed back to having 'gro off',
> shrug...
>
> daryl
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On 5/23/13 12:12 PM, daryl herzmann wrote:
> Thanks, it appears this is the relevant bug?
>
> https://bugzilla.redhat.com/show_bug.cgi?id=854066

Yep, that's it. Should be in the kernel you are running.
>
> I did a test of this:
>
> # ethtool -k eth1
> Features for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: off
>
> and the problem did not appear to go away. The traffic through the
> director was extremely sluggish, so I failed back to having 'gro off',
> shrug...
Did it get worse, or just no better?

Are you able to do iptables NAT instead to see if that makes a
difference, or perhaps put a http proxy in the middle? If nothing else,
just as a test. What about going direct to a http server - Does that
have better throughput?

David

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On Thu, 23 May 2013, David Coulson wrote:

>> and the problem did not appear to go away. The traffic through the
>> director was extremely sluggish, so I failed back to having 'gro off',
>> shrug...
> Did it get worse, or just no better?

Sorry, I was not very clear with my statement. I meant was that the old
bug of very poor / terrible performance still existed when GRO was on.
Turning if off leads to reasonable performance that I can live with :)

> Are you able to do iptables NAT instead to see if that makes a
> difference,

Sorry, ignorance is biting me here and I am not sure what you mean. I
thought iptables was necessary to make LVS-NAT work in the first place.

> or perhaps put a http proxy in the middle?

I do have other services in the mix than http, but I could try that.

> If nothing else, just as a test. What about going direct to a http
> server - Does that have better throughput?

Yeah, I can get gigabit transfer rates direct from a real server and
via the LVS-NAT for a single file on a single connection.

The LVS director seems to be maxing out around 50,000 packets per second
based on sar output.

Thanks for the help!

daryl

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On Thu, 2013-05-23 at 15:28 -0500, daryl herzmann wrote:
> Sorry, ignorance is biting me here and I am not sure what you mean. I
> thought iptables was necessary to make LVS-NAT work in the first place.

No, it isn't.

If you have rules in place and something making use of the conntrack
modules (matching ESTABLISHED/RELATED for example) then you *could* -
I'm not saying *will :) - see performance problems. That may explain the
"single connection is fast but lots at the same time aren't" scenario.
As the conntrack modules run in kernel space that could explain the CPU
usage stats, too.

Try turning off any conntrack-related rules and see if it helps.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On Fri, 24 May 2013, Graeme Fowler wrote:

> If you have rules in place and something making use of the conntrack
> modules (matching ESTABLISHED/RELATED for example) then you *could* -
> I'm not saying *will :) - see performance problems. That may explain the
> "single connection is fast but lots at the same time aren't" scenario.
> As the conntrack modules run in kernel space that could explain the CPU
> usage stats, too.
>
> Try turning off any conntrack-related rules and see if it helps.

Thanks for your help. Here's my current iptables setup, I thought I had
enabled NOTRACK for http and https traffic to prevent this. 1.1.1.1 is my
fake public IP for this email and 192.168.0.0 is the LAN.

Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain FORWARD (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination

Table: nat
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 SNAT all -- 192.168.0.0/24 0.0.0.0/0 to:1.1.1.1
2 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination

Table: mangle
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 MARK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:21 MARK set 0x15
# naughty machines blocked...
2 DROP all -- 1.1.1.77 0.0.0.0/0
3 DROP all -- 1.1.1.22 0.0.0.0/0

Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain FORWARD (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination

Table: raw
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:80
2 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp spt:80
3 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:443
4 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp spt:443

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:80
2 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp spt:80
3 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:443
4 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp spt:443

thank you,
daryl

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

daryl herzmann <akrherz@iastate.edu> wrote:

>On Fri, 24 May 2013, Graeme Fowler wrote:
>
>> If you have rules in place and something making use of the conntrack
>> modules (matching ESTABLISHED/RELATED for example) then you *could* -
>
>> I'm not saying *will :) - see performance problems. That may explain
>the
>> "single connection is fast but lots at the same time aren't"
>scenario.
>> As the conntrack modules run in kernel space that could explain the
>CPU
>> usage stats, too.
>>
>> Try turning off any conntrack-related rules and see if it helps.
>
>Thanks for your help. Here's my current iptables setup, I thought I
>had
>enabled NOTRACK for http and https traffic to prevent this. 1.1.1.1 is
>my
>fake public IP for this email and 192.168.0.0 is the LAN.
>
>Table: filter
>Chain INPUT (policy ACCEPT)
>num target prot opt source destination
>
>Chain FORWARD (policy ACCEPT)
>num target prot opt source destination
>
>Chain OUTPUT (policy ACCEPT)
>num target prot opt source destination
>
>Table: nat
>Chain PREROUTING (policy ACCEPT)
>num target prot opt source destination
>
>Chain POSTROUTING (policy ACCEPT)
>num target prot opt source destination
>1 SNAT all -- 192.168.0.0/24 0.0.0.0/0 to:1.1.1.1
>2 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
>
>Chain OUTPUT (policy ACCEPT)
>num target prot opt source destination
>
>Table: mangle
>Chain PREROUTING (policy ACCEPT)
>num target prot opt source destination
>1 MARK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:21 MARK set
>0x15
># naughty machines blocked...
>2 DROP all -- 1.1.1.77 0.0.0.0/0
>3 DROP all -- 1.1.1.22 0.0.0.0/0
>
>Chain INPUT (policy ACCEPT)
>num target prot opt source destination
>
>Chain FORWARD (policy ACCEPT)
>num target prot opt source destination
>
>Chain OUTPUT (policy ACCEPT)
>num target prot opt source destination
>
>Chain POSTROUTING (policy ACCEPT)
>num target prot opt source destination
>
>Table: raw
>Chain PREROUTING (policy ACCEPT)
>num target prot opt source destination
>1 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:80
>2 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp
>spt:80
>3 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:443
>4 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp
>spt:443
>
>Chain OUTPUT (policy ACCEPT)
>num target prot opt source destination
>1 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:80
>2 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp
>spt:80
>3 NOTRACK tcp -- 0.0.0.0/0 1.1.1.1 tcp dpt:443
>4 NOTRACK tcp -- 192.168.0.0/24 0.0.0.0/0 tcp
>spt:443
>
>thank you,
> daryl
>
>_______________________________________________
>Please read the documentation before posting - it's available at:
>http://www.linuxvirtualserver.org/
>
>LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>Send requests to lvs-users-request@LinuxVirtualServer.org
>or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Have you tried running a test with no iptables rules at all?

Even NOTRACK has to parse the packet headers before handing it to IPVS, so there's obviously a performance hit there.

Graeme
[sent from mobile, excuse top-posting]

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On Sat, 25 May 2013, Graeme Fowler wrote:

> Have you tried running a test with no iptables rules at all?

Unfortunately, it would be difficult for me to test that as I need SNAT
working for my various applications to keep working. I think my next step
will be to try a different machine with a current Xeon processor and beefy
NIC.

Thanks again for the help with this. I feel a bit more confident now that
I don't have an aggregious error with my setup.

daryl

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On Thu, 23 May 2013, daryl herzmann wrote:

>> On Thu, 23 May 2013, David Coulson wrote:
>>
>> I was told by RedHat the GRO issue is fixed in RHEL6.4 - Not had chance to
>> test it yet.
>
> Thanks, it appears this is the relevant bug?
>
> https://bugzilla.redhat.com/show_bug.cgi?id=854066

To follow up on this, I opened a Red Hat bugzilla ticket (#973190, sorry
it's private) regarding the problems I was seeing with GRO and LVS NAT.
After some debugging, they found the issue and the fix should be in the
RHEL 6.5 kernel (bz #973122, sorry private as well).

daryl

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Hi,

> To follow up on this, I opened a Red Hat bugzilla ticket (#973190, sorry
> it's private) regarding the problems I was seeing with GRO and LVS NAT.
> After some debugging, they found the issue and the fix should be in the
> RHEL 6.5 kernel (bz #973122, sorry private as well).

Did they mention if this patch will be propagated upstream? It's affecting at least the Debian 6 2.6.32 kernel, too.

TIA,
Thomas



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Hi,

On Tue, 9 Jul 2013, Thomas Bätzler wrote:

> Hi,
>
>> To follow up on this, I opened a Red Hat bugzilla ticket (#973190, sorry
>> it's private) regarding the problems I was seeing with GRO and LVS NAT.
>> After some debugging, they found the issue and the fix should be in the
>> RHEL 6.5 kernel (bz #973122, sorry private as well).
>
> Did they mention if this patch will be propagated upstream? It's
> affecting at least the Debian 6 2.6.32 kernel, too.

They did not make any mention of upstreaming.

daryl

On Tue, Jul 09, 2013 at 07:52:38AM -0500, daryl herzmann wrote:
> Hi,
>
> On Tue, 9 Jul 2013, Thomas Bätzler wrote:
>
> >Hi,
> >
> >>To follow up on this, I opened a Red Hat bugzilla ticket (#973190, sorry
> >>it's private) regarding the problems I was seeing with GRO and LVS NAT.
> >>After some debugging, they found the issue and the fix should be in the
> >>RHEL 6.5 kernel (bz #973122, sorry private as well).
> >
> >Did they mention if this patch will be propagated upstream? It's
> >affecting at least the Debian 6 2.6.32 kernel, too.
>
> They did not make any mention of upstreaming.
>
> daryl

Bugzilla says "Upstream commit 8f1b03a4c18e8f3f0801447b62330faa8ed3bb37
fixes this.".

But perhaps that is only a portion of the fix?


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users