Mailing List Archive: TCP v UDP performance

TCP v UDP performance

Jun 11, 2002, 6:04 AM

Post #1 of 6 (4873 views)

*** Before acting on this e-mail or opening any attachment you are advised to read the disclaimer at the end of this e-mail ***

Sometime ago I posted some performance tweaks for TCP performance, which
(when the filer was switched to TCP) worked a treat. I was hoping to get
the same for UDP. However...

It seems that UDP on Sun over 1GB Ethernet and above is problematic. Some
details below (words and details have been changed to protect the guilty!).
As a result we're sticking to TCP for now.

Peter

=====================================================

I've done a bit of research and found out that testing of a very similar
setup to yours, by another customer, showed that there is a performance bug
in the NFS UDP implementation of Solaris.

We all know that NFS over UDP is about 20% faster than NFS over TCP however,
the Solaris implementation of UDP breaks this rule when gigabit Ethernet is
involved..! It seems to be that when NFS is used with UDP for internet,
10Mb and 100Mb connectivity then the rule applies and also when you have a
gigabit Ethernet connection under low load then again all is fine.

The performance problem appears when you try to load the gig connection
heavily - Solaris's UDP simply cannot perform at the required rate. Now,
thankfully, the TCP implementation in Solaris is very good and is up to the
job - hence, the situation you find whereby NFS over TCP is much better than
NFS over UDP. This matches exactly with your findings where under normal
database usage your Gig + NFS/UDP connection is fine - but when you try to
backup over the Gig Ethernet link the load on the network goes up and UDP
can't perform. You switched from UDP to TCP and the problem went away.

My recommendation to you is to keep using NFS over TCP - the other customer
has decided to stay with TCP as it works well and they can't see any benefit
in moving back to UDP.

******************************* Disclaimer *****************************
Confidentiality: This e-mail and any attachments are intended for the
addressee(s) only and may be confidential. If they have come to you in
error you must take no action based on them, nor must you copy or show
them to anyone; please reply to this e-mail and highlight the error.

Viruses: Although we have taken steps to ensure that this e-mail and
attachments are free from any virus, we advise that in keeping with good
practice the recipient should ensure they are actually virus free.
********************** http://www.burallplastec.com ********************

Re: TCP v UDP performance [ In reply to ]

cash at UDel

Jun 12, 2002, 11:01 AM

Post #2 of 6 (4735 views)

Permalink

On Tue, 11 Jun 2002, Peter Bryant wrote:

> Sometime ago I posted some performance tweaks for TCP performance, which
> (when the filer was switched to TCP) worked a treat. I was hoping to get
> the same for UDP. However...
>
> It seems that UDP on Sun over 1GB Ethernet and above is problematic. Some
> details below (words and details have been changed to protect the guilty!).
> As a result we're sticking to TCP for now.

What version of solaris and what gig-e card? I would look for a new
driver for the gigaswift adaptor that helps immensly w/ UDP NFS perf.

Jason

>
> Peter
>
> =====================================================
>
> I've done a bit of research and found out that testing of a very similar
> setup to yours, by another customer, showed that there is a performance bug
> in the NFS UDP implementation of Solaris.
>
> We all know that NFS over UDP is about 20% faster than NFS over TCP however,
> the Solaris implementation of UDP breaks this rule when gigabit Ethernet is
> involved..! It seems to be that when NFS is used with UDP for internet,
> 10Mb and 100Mb connectivity then the rule applies and also when you have a
> gigabit Ethernet connection under low load then again all is fine.
>
> The performance problem appears when you try to load the gig connection
> heavily - Solaris's UDP simply cannot perform at the required rate. Now,
> thankfully, the TCP implementation in Solaris is very good and is up to the
> job - hence, the situation you find whereby NFS over TCP is much better than
> NFS over UDP. This matches exactly with your findings where under normal
> database usage your Gig + NFS/UDP connection is fine - but when you try to
> backup over the Gig Ethernet link the load on the network goes up and UDP
> can't perform. You switched from UDP to TCP and the problem went away.
>
> My recommendation to you is to keep using NFS over TCP - the other customer
> has decided to stay with TCP as it works well and they can't see any benefit
> in moving back to UDP.
>
>
> ******************************* Disclaimer *****************************
> Confidentiality: This e-mail and any attachments are intended for the
> addressee(s) only and may be confidential. If they have come to you in
> error you must take no action based on them, nor must you copy or show
> them to anyone; please reply to this e-mail and highlight the error.
>
> Viruses: Although we have taken steps to ensure that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> practice the recipient should ensure they are actually virus free.
> ********************** http://www.burallplastec.com ********************
>

/* Jason S. Cash - Systems Programmer
IT/Network and Systems Services
University of Delaware, Newark Delaware
e:cash@udel.edu v: 302-831-0461 */

RE: TCP v UDP performance [ In reply to ]

PBryant at burallplastec

Jun 13, 2002, 3:37 AM

Post #3 of 6 (4749 views)

Permalink

*** Before acting on this e-mail or opening any attachment you are advised to read the disclaimer at the end of this e-mail ***

Solaris 6, and we used Sun's own PCI Gigabit 2.0 card (as it's described on
the invoice)

Thanks for the tip though. I suspect the drivers are probably OK, install
was only done a couple of months back and up to date versions used - unless
you know a previous driver is better.... <g>

Peter

-----Original Message-----
From: Jason S. Cash [mailto:cash@UDel.Edu]
Sent: 12 June 2002 19:02
To: Peter Bryant
Cc: 'toasters@mathworks.com'
Subject: Re: TCP v UDP performance

On Tue, 11 Jun 2002, Peter Bryant wrote:

> Sometime ago I posted some performance tweaks for TCP performance,
> which (when the filer was switched to TCP) worked a treat. I was
> hoping to get the same for UDP. However...
>
> It seems that UDP on Sun over 1GB Ethernet and above is problematic.
> Some details below (words and details have been changed to protect the
> guilty!). As a result we're sticking to TCP for now.

What version of solaris and what gig-e card? I would look for a new driver
for the gigaswift adaptor that helps immensly w/ UDP NFS perf.

Jason

>
> Peter
>
> =====================================================
>
> I've done a bit of research and found out that testing of a very
> similar setup to yours, by another customer, showed that there is a
> performance bug in the NFS UDP implementation of Solaris.
>
> We all know that NFS over UDP is about 20% faster than NFS over TCP
> however, the Solaris implementation of UDP breaks this rule when
> gigabit Ethernet is involved..! It seems to be that when NFS is used
> with UDP for internet, 10Mb and 100Mb connectivity then the rule
> applies and also when you have a gigabit Ethernet connection under low
> load then again all is fine.
>
> The performance problem appears when you try to load the gig
> connection heavily - Solaris's UDP simply cannot perform at the
> required rate. Now, thankfully, the TCP implementation in Solaris is
> very good and is up to the job - hence, the situation you find whereby
> NFS over TCP is much better than NFS over UDP. This matches exactly
> with your findings where under normal database usage your Gig +
> NFS/UDP connection is fine - but when you try to backup over the Gig
> Ethernet link the load on the network goes up and UDP can't perform.
> You switched from UDP to TCP and the problem went away.
>
> My recommendation to you is to keep using NFS over TCP - the other
> customer has decided to stay with TCP as it works well and they can't
> see any benefit in moving back to UDP.
>
>
> ******************************* Disclaimer
> *****************************
> Confidentiality: This e-mail and any attachments are intended for the
> addressee(s) only and may be confidential. If they have come to you in
> error you must take no action based on them, nor must you copy or show
> them to anyone; please reply to this e-mail and highlight the error.
>
> Viruses: Although we have taken steps to ensure that this e-mail and
> attachments are free from any virus, we advise that in keeping with
> good practice the recipient should ensure they are actually virus
> free.
> ********************** http://www.burallplastec.com ********************
>

/* Jason S. Cash - Systems Programmer
IT/Network and Systems Services
University of Delaware, Newark Delaware
e:cash@udel.edu v: 302-831-0461 */

RE: TCP v UDP performance [ In reply to ]

kgraham at dotnetdotcom

Jun 13, 2002, 10:01 PM

Post #4 of 6 (4754 views)

Permalink

> Thanks for the tip though. I suspect the drivers are probably OK, install
> was only done a couple of months back and up to date versions used - unless
> you know a previous driver is better.... <g>

For 2.6, do make sure you have 106764-08 installed. Since SUNWged is an
unbundled driver, patches for it will _not_ be included in 'Recommended'
clusters (one of the reasons I despise them), nor will patchdiag include
it (unless its logic has been altered significantly since I last read
source).

I didn't take the time to read all the bug reports, but there were
resolved performance issues with ttcp workloads which will resemble udp to
the driver since IIRC that card doesn't do anything like tcp checksum
offloading.

..kg..

. o O ( still wondering what was so wrong with the rebranded Alteon ACEnic
Sun Gigabit Ethernet 1.0 that they stuck us with this crap. Do yourself a
favor and buy a SysKonnect or IntraServer card)

> -----Original Message-----
> From: Jason S. Cash [mailto:cash@UDel.Edu]
> Sent: 12 June 2002 19:02
> To: Peter Bryant
> Cc: 'toasters@mathworks.com'
> Subject: Re: TCP v UDP performance
>
>
> On Tue, 11 Jun 2002, Peter Bryant wrote:
>
> > Sometime ago I posted some performance tweaks for TCP performance,
> > which (when the filer was switched to TCP) worked a treat. I was
> > hoping to get the same for UDP. However...
> >
> > It seems that UDP on Sun over 1GB Ethernet and above is problematic.
> > Some details below (words and details have been changed to protect the
> > guilty!). As a result we're sticking to TCP for now.
>
> What version of solaris and what gig-e card? I would look for a new driver
> for the gigaswift adaptor that helps immensly w/ UDP NFS perf.
>
> Jason
>
>
> >
> > Peter
> >
> > =====================================================
> >
> > I've done a bit of research and found out that testing of a very
> > similar setup to yours, by another customer, showed that there is a
> > performance bug in the NFS UDP implementation of Solaris.
> >
> > We all know that NFS over UDP is about 20% faster than NFS over TCP
> > however, the Solaris implementation of UDP breaks this rule when
> > gigabit Ethernet is involved..! It seems to be that when NFS is used
> > with UDP for internet, 10Mb and 100Mb connectivity then the rule
> > applies and also when you have a gigabit Ethernet connection under low
> > load then again all is fine.
> >
> > The performance problem appears when you try to load the gig
> > connection heavily - Solaris's UDP simply cannot perform at the
> > required rate. Now, thankfully, the TCP implementation in Solaris is
> > very good and is up to the job - hence, the situation you find whereby
> > NFS over TCP is much better than NFS over UDP. This matches exactly
> > with your findings where under normal database usage your Gig +
> > NFS/UDP connection is fine - but when you try to backup over the Gig
> > Ethernet link the load on the network goes up and UDP can't perform.
> > You switched from UDP to TCP and the problem went away.
> >
> > My recommendation to you is to keep using NFS over TCP - the other
> > customer has decided to stay with TCP as it works well and they can't
> > see any benefit in moving back to UDP.
> >
> >
> > ******************************* Disclaimer
> > *****************************
> > Confidentiality: This e-mail and any attachments are intended for the
> > addressee(s) only and may be confidential. If they have come to you in
> > error you must take no action based on them, nor must you copy or show
> > them to anyone; please reply to this e-mail and highlight the error.
> >
> > Viruses: Although we have taken steps to ensure that this e-mail and
> > attachments are free from any virus, we advise that in keeping with
> > good practice the recipient should ensure they are actually virus
> > free.
> > ********************** http://www.burallplastec.com ********************
> >
>
> /* Jason S. Cash - Systems Programmer
> IT/Network and Systems Services
> University of Delaware, Newark Delaware
> e:cash@udel.edu v: 302-831-0461 */
>

RE: TCP v UDP performance [ In reply to ]

mds at gbnet

Jun 14, 2002, 3:17 AM

Post #5 of 6 (4754 views)

Permalink

On Thu 13 Jun, 2002, kevin graham <kgraham@dotnetdotcom.org> wrote:
>
> For 2.6, do make sure you have 106764-08 installed. Since SUNWged is an
> unbundled driver, patches for it will _not_ be included in 'Recommended'
> clusters (one of the reasons I despise them), nor will patchdiag include
> it (unless its logic has been altered significantly since I last read
> source).

Yeah, the patchdiag and patchcheck scripts will believe what they're
told in the .xref. However, I always use the '-l' flag which appends a
list of patches worth considering based on your other installed packages.

If you'd like the output to be in html with links to the patch pages in
sunsolve, then take a look at http://www.gbnet.net/~mds/pd.pl . There's
an extra switch '-w' which makes it stuff random bits of HTML into
the output and I find it saves me hours when assembling bespoke patch
lists. For the US sunsolve use '-W'.

The output is way faster for the browser to handle than that from
patchcheck and in some ways more useful.

Needless to say this is a complete hack, is unsupported by everyone, is
a mild abuse of Sun copyright and I should probably be hung by the RIAA
and the Fatherland Defence Force for even thinking about it. But if it's
useful then I hope I earn some forgiveness along the way.

>-- End of excerpt from kevin graham

--
-Mark ... an Englishman in London ...

RE: TCP v UDP performance [ In reply to ]

PBryant at burallplastec

Jun 14, 2002, 5:37 AM

Post #6 of 6 (4725 views)

Permalink

*** Before acting on this e-mail or opening any attachment you are advised to read the disclaimer at the end of this e-mail ***

Thanks for the tip Kevin (and others),

as we've a managed solution for the Sun kit I'll pass this on them as knows
(and who should have got the damn thing working properly in the first
place).

Strange that with only a modest amount of filer and Solaris knowledge (but
of course access to this list) that I ended up finding the information that
got the performance sorted instead of the paid for professionals <g>

Thanks toaster world!

Peter

-----Original Message-----
From: kevin graham [mailto:kgraham@dotnetdotcom.org]
Sent: 14 June 2002 06:02
To: Peter Bryant
Cc: 'toasters@mathworks.com'
Subject: RE: TCP v UDP performance

> Thanks for the tip though. I suspect the drivers are probably OK,
> install was only done a couple of months back and up to date versions
> used - unless you know a previous driver is better.... <g>

For 2.6, do make sure you have 106764-08 installed. Since SUNWged is an
unbundled driver, patches for it will _not_ be included in 'Recommended'
clusters (one of the reasons I despise them), nor will patchdiag include it
(unless its logic has been altered significantly since I last read source).

I didn't take the time to read all the bug reports, but there were resolved
performance issues with ttcp workloads which will resemble udp to
the driver since IIRC that card doesn't do anything like tcp checksum
offloading.

..kg..

. o O ( still wondering what was so wrong with the rebranded Alteon ACEnic
Sun Gigabit Ethernet 1.0 that they stuck us with this crap. Do yourself a
favor and buy a SysKonnect or IntraServer card)

> -----Original Message-----
> From: Jason S. Cash [mailto:cash@UDel.Edu]
> Sent: 12 June 2002 19:02
> To: Peter Bryant
> Cc: 'toasters@mathworks.com'
> Subject: Re: TCP v UDP performance
>
>
> On Tue, 11 Jun 2002, Peter Bryant wrote:
>
> > Sometime ago I posted some performance tweaks for TCP performance,
> > which (when the filer was switched to TCP) worked a treat. I was
> > hoping to get the same for UDP. However...
> >
> > It seems that UDP on Sun over 1GB Ethernet and above is problematic.
> > Some details below (words and details have been changed to protect the
> > guilty!). As a result we're sticking to TCP for now.
>
> What version of solaris and what gig-e card? I would look for a new
> driver for the gigaswift adaptor that helps immensly w/ UDP NFS perf.
>
> Jason
>
>
> >
> > Peter
> >
> > =====================================================
> >
> > I've done a bit of research and found out that testing of a very
> > similar setup to yours, by another customer, showed that there is a
> > performance bug in the NFS UDP implementation of Solaris.
> >
> > We all know that NFS over UDP is about 20% faster than NFS over TCP
> > however, the Solaris implementation of UDP breaks this rule when
> > gigabit Ethernet is involved..! It seems to be that when NFS is used
> > with UDP for internet, 10Mb and 100Mb connectivity then the rule
> > applies and also when you have a gigabit Ethernet connection under low
> > load then again all is fine.
> >
> > The performance problem appears when you try to load the gig
> > connection heavily - Solaris's UDP simply cannot perform at the
> > required rate. Now, thankfully, the TCP implementation in Solaris is
> > very good and is up to the job - hence, the situation you find whereby
> > NFS over TCP is much better than NFS over UDP. This matches exactly
> > with your findings where under normal database usage your Gig +
> > NFS/UDP connection is fine - but when you try to backup over the Gig
> > Ethernet link the load on the network goes up and UDP can't perform.
> > You switched from UDP to TCP and the problem went away.
> >
> > My recommendation to you is to keep using NFS over TCP - the other
> > customer has decided to stay with TCP as it works well and they can't
> > see any benefit in moving back to UDP.
> >
> >
> > ******************************* Disclaimer
> > *****************************
> > Confidentiality: This e-mail and any attachments are intended for the
> > addressee(s) only and may be confidential. If they have come to you in
> > error you must take no action based on them, nor must you copy or show
> > them to anyone; please reply to this e-mail and highlight the error.
> >
> > Viruses: Although we have taken steps to ensure that this e-mail
> > and
> > attachments are free from any virus, we advise that in keeping with
> > good practice the recipient should ensure they are actually virus
> > free.
> > ********************** http://www.burallplastec.com ********************
> >
>
> /* Jason S. Cash - Systems Programmer
> IT/Network and Systems Services
> University of Delaware, Newark Delaware
> e:cash@udel.edu v: 302-831-0461 */
>