Mailing List Archive

Hi

Zhongkang Lao wrote on 05.12.2012 08:41:
> I have already googled this like a hundred of times, but I got nothing.
> I didn't find any page or any article explain this.
> I'm sure someone here can explain this.
> So sorry to bother you guys here.

I don't have any answers, but I can confirm the problem (and the
workaround) on a box running Debian "Squeeze" with 2.6.32 directing
traffic from a VIP bound to lo via NAT to an active/backup bonding
interface on the backend. With "gro on" for the bonded interfaces to the
backend real servers, throughput was ~64K/s. With "gro off" it was
~50MB/s. It's not a general networking problem since a direct connection
to the real server via a DNAT port forward was not impacted.

>From what I gleaned from a cursory look with tcpdump it seems that the
problem is caused by incorrectly calculated tcp checksums.

Anybody interested in pursuing this further?


Cheers,
Thomas


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Hello,

On Tue, 11 Dec 2012, Thomas Bätzler wrote:

> Hi
>
> Zhongkang Lao wrote on 05.12.2012 08:41:
> > I have already googled this like a hundred of times, but I got nothing.
> > I didn't find any page or any article explain this.
> > I'm sure someone here can explain this.
> > So sorry to bother you guys here.
>
> I don't have any answers, but I can confirm the problem (and the
> workaround) on a box running Debian "Squeeze" with 2.6.32 directing
> traffic from a VIP bound to lo via NAT to an active/backup bonding
> interface on the backend. With "gro on" for the bonded interfaces to the
> backend real servers, throughput was ~64K/s. With "gro off" it was
> ~50MB/s. It's not a general networking problem since a direct connection
> to the real server via a DNAT port forward was not impacted.
>
> >From what I gleaned from a cursory look with tcpdump it seems that the
> problem is caused by incorrectly calculated tcp checksums.
>
> Anybody interested in pursuing this further?

It is fixed in mainline more than 2 years ago:

===
commit 8f1b03a4c18e8f3f0801447b62330faa8ed3bb37
Author: Simon Horman <horms@verge.net.au>
Date: Tue Nov 9 10:08:49 2010 +0900

ipvs: allow transmit of GRO aggregated skbs

Attempt at allowing LVS to transmit skbs of greater than MTU length that
have been aggregated by GRO and can thus be deaggregated by GSO.
===

Without this fix IPVS assumes that large packets
are received and sends FRAG_NEEDED messages. For old
kernels GRO should be disabled to avoid the problem.

> Cheers,
> Thomas

Regards

--
Julian Anastasov <ja@ssi.bg>

Le 11/12/2012 22:00, Thomas Bätzler a écrit :
> Hi
>
> Zhongkang Lao wrote on 05.12.2012 08:41:
>> I have already googled this like a hundred of times, but I got nothing.
>> I didn't find any page or any article explain this.
>> I'm sure someone here can explain this.
>> So sorry to bother you guys here.
>
> I don't have any answers, but I can confirm the problem (and the
> workaround) on a box running Debian "Squeeze" with 2.6.32 directing
> traffic from a VIP bound to lo via NAT to an active/backup bonding
> interface on the backend. With "gro on" for the bonded interfaces to the
> backend real servers, throughput was ~64K/s. With "gro off" it was
> ~50MB/s. It's not a general networking problem since a direct connection
> to the real server via a DNAT port forward was not impacted.
>
>>From what I gleaned from a cursory look with tcpdump it seems that the
> problem is caused by incorrectly calculated tcp checksums.
>
> Anybody interested in pursuing this further?
>

Yes, I can relate.
I had the same conclusions, incorrectly calculated TCP checksums leading
to packet drops by the next firewall (in my architecture).
Also fixed by disabling gro.
CentOS 6.2, ipvsadm-1.25-9.el6.x86_64, kernel 2.6.32-220.23.1.el6.x86_64



--
Cheers,
Florian Crouzat

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On Wed, 2012-12-12 at 10:05 +0100, Florian Crouzat wrote:
> Le 11/12/2012 22:00, Thomas Bätzler a écrit :
[...]
> >
> > I don't have any answers, but I can confirm the problem (and the
> > workaround) on a box running Debian "Squeeze" with 2.6.32 directing
> > traffic from a VIP bound to lo via NAT to an active/backup bonding
> > interface on the backend. With "gro on" for the bonded interfaces to the
> > backend real servers, throughput was ~64K/s. With "gro off" it was
> > ~50MB/s. It's not a general networking problem since a direct connection
> > to the real server via a DNAT port forward was not impacted.
> >
> >>From what I gleaned from a cursory look with tcpdump it seems that the
> > problem is caused by incorrectly calculated tcp checksums.
> >
> > Anybody interested in pursuing this further?

As Julian notes, this have already been fixed in upstream kernels.
(http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8f1b03a4c18e8f3f0801447b62330faa8ed3bb37)

And I have recently (Aug 2012) also fixed this for IPv6.
(http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=590e3f79a21edd2e9857ac3ced25ba6b2a491ef8)


> Yes, I can relate.
> I had the same conclusions, incorrectly calculated TCP checksums leading
> to packet drops by the next firewall (in my architecture).
> Also fixed by disabling gro.

Its not related to incorrect checksums, but (as you noticed) the GRO
(Generic Receive Offload) feature. (Adjusted via ethtool -K ethX)


> CentOS 6.2, ipvsadm-1.25-9.el6.x86_64, kernel 2.6.32-220.23.1.el6.x86_64

FYI: I have already backported support (for GRO) for the upcoming
RHEL6.4 and RHEL5.9 kernels. (Its also in zstream for 5.8 in
kernel-2.6.18-308.19.1.el5)

To be exact, this fixed in kernel releases (and beyond):
RHEL6 kernel-2.6.32-328.el6
RHEL5 kernel-2.6.18-340.el5

If someone needs this before its officially part of a RHEL release, I
think I'm allowed to give you the kernel with this fix as a test kernel.
Just let me now directly.

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Sr. Network Kernel Developer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

So the reason is that LVS cann't transmit skbs of greater than MTU
that have been aggregated by GRO.
And Simon Horman gived a patch to make LVS deaggregate skbs that's greater
than MTU by GSO.
(
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8f1b03a4c18e8f3f0801447b62330faa8ed3bb37
)

Great! Thank you very much! :)


On Sat, Dec 15, 2012 at 12:38 AM, Jesper Dangaard Brouer
<brouer@redhat.com>wrote:

> On Wed, 2012-12-12 at 10:05 +0100, Florian Crouzat wrote:
> > Le 11/12/2012 22:00, Thomas Bätzler a écrit :
> [...]
> > >
> > > I don't have any answers, but I can confirm the problem (and the
> > > workaround) on a box running Debian "Squeeze" with 2.6.32 directing
> > > traffic from a VIP bound to lo via NAT to an active/backup bonding
> > > interface on the backend. With "gro on" for the bonded interfaces to
> the
> > > backend real servers, throughput was ~64K/s. With "gro off" it was
> > > ~50MB/s. It's not a general networking problem since a direct
> connection
> > > to the real server via a DNAT port forward was not impacted.
> > >
> > >>From what I gleaned from a cursory look with tcpdump it seems that the
> > > problem is caused by incorrectly calculated tcp checksums.
> > >
> > > Anybody interested in pursuing this further?
>
> As Julian notes, this have already been fixed in upstream kernels.
> (
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8f1b03a4c18e8f3f0801447b62330faa8ed3bb37
> )
>
> And I have recently (Aug 2012) also fixed this for IPv6.
> (
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=590e3f79a21edd2e9857ac3ced25ba6b2a491ef8
> )
>
>
> > Yes, I can relate.
> > I had the same conclusions, incorrectly calculated TCP checksums leading
> > to packet drops by the next firewall (in my architecture).
> > Also fixed by disabling gro.
>
> Its not related to incorrect checksums, but (as you noticed) the GRO
> (Generic Receive Offload) feature. (Adjusted via ethtool -K ethX)
>
>
> > CentOS 6.2, ipvsadm-1.25-9.el6.x86_64, kernel 2.6.32-220.23.1.el6.x86_64
>
> FYI: I have already backported support (for GRO) for the upcoming
> RHEL6.4 and RHEL5.9 kernels. (Its also in zstream for 5.8 in
> kernel-2.6.18-308.19.1.el5)
>
> To be exact, this fixed in kernel releases (and beyond):
> RHEL6 kernel-2.6.32-328.el6
> RHEL5 kernel-2.6.18-340.el5
>
> If someone needs this before its officially part of a RHEL release, I
> think I'm allowed to give you the kernel with this fix as a test kernel.
> Just let me now directly.
>
> --
> Best regards,
> Jesper Dangaard Brouer
> MSc.CS, Sr. Network Kernel Developer at Red Hat
> Author of http://www.iptv-analyzer.org
> LinkedIn: http://www.linkedin.com/in/brouer
>
>
>


--
Stephen Lao
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users