Mailing List Archive: Sun Solaris 8 optimal performance with F880

Sun Solaris 8 optimal performance with F880

Sep 19, 2002, 6:17 PM

Post #1 of 9 (3084 views)

Toasters

Configuration : Sun E6000 server running Solaris 8 latest patch level
Filer : F880
Cisco Switch 6509(Flowcontrol turned on)

We have recently bought 2- F880 filers and we are in the implementation phases. The weird
performance results we are getting with following system settings and mount parameters.
The output from "sysstat 1" on filer, shows writes 35 MBPS but reads just very slow
or not even generate any traffic. Here are the settings

ndd -set /dev/udp udp_recv_hiwat 65535
ndd -set /dev/udp udp_xmit_hiwat 65535
ndd -set /dev/tcp tcp_recv_hiwat 65535
ndd -set /dev/tcp tcp_xmit_hiwat 65535
ndd -set /dev/ge instance 1
ndd -set /dev/ge adv_pauseTX 1
ndd -set /dev/ge adv_pauseRX 1
ndd -set /dev/ge adv_1000autoneg_cap 0
ndd -set /dev/ge adv_1000fdx_cap 1

set sq_max_size = 30
set nstrpush = 90
set ncsize 8000
set nfs:nfs3_max_threads = 24
set nfs:nfs3_nra =10
set nfs:nfs_max_threads = 24
set nfs:nfs_nra = 10

Mount file system :

mount -o rw,bg,hard,intr,proto=udp,vers=3,rsize=32768,wsize=32768 filer01:/vol/test /test

Test method

dd if=/dev/zero of=/test/foo bs=32k count=50000

I have tried to search some documents on now site for the optimal performance but I could not
find any optimal numbers which I should try to achieve.
Why reads are chocking up ?
anyone has any proven configuration which works optimally ?

Thanks in advance for your inputs and thoughts.

Regards

-Deepak Soneji
Flextronics International

Re: Sun Solaris 8 optimal performance with F880 [ In reply to ]

brilong at cisco

Sep 19, 2002, 6:47 PM

Post #2 of 9 (3061 views)

Permalink

UDP and 32K transfers don't mix. Reduce to 8K transfers (nfs.udp.xfersize
8192) or change to TCP, the preferred method in which to use NFS.

/Brian/

> Toasters
>
> Configuration : Sun E6000 server running Solaris 8 latest patch level
> Filer : F880
> Cisco Switch 6509(Flowcontrol turned on)
>
>
> We have recently bought 2- F880 filers and we are in the implementation phases. The weird
> performance results we are getting with following system settings and mount parameters.
> The output from "sysstat 1" on filer, shows writes 35 MBPS but reads just very slow
> or not even generate any traffic. Here are the settings
>
> ndd -set /dev/udp udp_recv_hiwat 65535
> ndd -set /dev/udp udp_xmit_hiwat 65535
> ndd -set /dev/tcp tcp_recv_hiwat 65535
> ndd -set /dev/tcp tcp_xmit_hiwat 65535
> ndd -set /dev/ge instance 1
> ndd -set /dev/ge adv_pauseTX 1
> ndd -set /dev/ge adv_pauseRX 1
> ndd -set /dev/ge adv_1000autoneg_cap 0
> ndd -set /dev/ge adv_1000fdx_cap 1
>
> set sq_max_size = 30
> set nstrpush = 90
> set ncsize 8000
> set nfs:nfs3_max_threads = 24
> set nfs:nfs3_nra =10
> set nfs:nfs_max_threads = 24
> set nfs:nfs_nra = 10
>
> Mount file system :
>
> mount -o rw,bg,hard,intr,proto=udp,vers=3,rsize=32768,wsize=32768 filer01:/vol/test /test
>
> Test method
>
> dd if=/dev/zero of=/test/foo bs=32k count=50000
>
>
> I have tried to search some documents on now site for the optimal performance but I could not
> find any optimal numbers which I should try to achieve.
> Why reads are chocking up ?
> anyone has any proven configuration which works optimally ?
>
> Thanks in advance for your inputs and thoughts.
>
> Regards
>
> -Deepak Soneji
> Flextronics International
>
>

--
Brian Long | | |
Americas IT Hosting Sys Admin | .|||. .|||.
Phone: (919) 392-7363 | ..:|||||||:...:|||||||:..
Pager: (888) 651-2015 | C i s c o S y s t e m s

Re: Sun Solaris 8 optimal performance with F880 [ In reply to ]

sonejideepak at hotmail

Sep 19, 2002, 11:04 PM

Post #3 of 9 (3042 views)

Permalink

Brian/Tim

Great ! READS are greatly improved upto 12-15 mbps after changing xfersize
to 8k.
Any idea about what magic numbers for read/write I should try to achieve
while tuning ?

Another question, why TCP ?

Thanks
-Deepak
----- Original Message -----
From: "Brian Long" <brilong@cisco.com>
To: "Deepak Soneji" <sonejideepak@hotmail.com>
Cc: <toasters@mathworks.com>
Sent: Thursday, September 19, 2002 6:47 PM
Subject: Re: Sun Solaris 8 optimal performance with F880

> UDP and 32K transfers don't mix. Reduce to 8K transfers (nfs.udp.xfersize
> 8192) or change to TCP, the preferred method in which to use NFS.
>
> /Brian/
>
> > Toasters
> >
> > Configuration : Sun E6000 server running Solaris 8 latest patch level
> > Filer : F880
> > Cisco Switch 6509(Flowcontrol turned on)
> >
> >
> > We have recently bought 2- F880 filers and we are in the implementation
phases. The weird
> > performance results we are getting with following system settings and
mount parameters.
> > The output from "sysstat 1" on filer, shows writes 35 MBPS but reads
just very slow
> > or not even generate any traffic. Here are the settings
> >
> > ndd -set /dev/udp udp_recv_hiwat 65535
> > ndd -set /dev/udp udp_xmit_hiwat 65535
> > ndd -set /dev/tcp tcp_recv_hiwat 65535
> > ndd -set /dev/tcp tcp_xmit_hiwat 65535
> > ndd -set /dev/ge instance 1
> > ndd -set /dev/ge adv_pauseTX 1
> > ndd -set /dev/ge adv_pauseRX 1
> > ndd -set /dev/ge adv_1000autoneg_cap 0
> > ndd -set /dev/ge adv_1000fdx_cap 1
> >
> > set sq_max_size = 30
> > set nstrpush = 90
> > set ncsize 8000
> > set nfs:nfs3_max_threads = 24
> > set nfs:nfs3_nra =10
> > set nfs:nfs_max_threads = 24
> > set nfs:nfs_nra = 10
> >
> > Mount file system :
> >
> > mount -o rw,bg,hard,intr,proto=udp,vers=3,rsize=32768,wsize=32768
filer01:/vol/test /test
> >
> > Test method
> >
> > dd if=/dev/zero of=/test/foo bs=32k count=50000
> >
> >
> > I have tried to search some documents on now site for the optimal
performance but I could not
> > find any optimal numbers which I should try to achieve.
> > Why reads are chocking up ?
> > anyone has any proven configuration which works optimally ?
> >
> > Thanks in advance for your inputs and thoughts.
> >
> > Regards
> >
> > -Deepak Soneji
> > Flextronics International
> >
> >
>
> --
> Brian Long | | |
> Americas IT Hosting Sys Admin | .|||. .|||.
> Phone: (919) 392-7363 | ..:|||||||:...:|||||||:..
> Pager: (888) 651-2015 | C i s c o S y s t e m s
>

RE: Sun Solaris 8 optimal performance with F880 [ In reply to ]

art at arzoon

Sep 20, 2002, 11:32 AM

Post #4 of 9 (3049 views)

Permalink

I thought the 880's were supposed to get around 60MB writes. Seems that
12-15 would be
worse than an F760.

Is there any published numbers for read or write performance for the
various models? I think
the licensing forbids Netapp users from discussing performance numbers for
some reason, am I
mistaken on this?

art

-----Original Message-----
From: Deepak Soneji [mailto:sonejideepak@hotmail.com]
Sent: Thursday, September 19, 2002 11:05 PM
To: Brian Long
Cc: toasters@mathworks.com
Subject: Re: Sun Solaris 8 optimal performance with F880

Brian/Tim

Great ! READS are greatly improved upto 12-15 mbps after changing xfersize
to 8k.
Any idea about what magic numbers for read/write I should try to achieve
while tuning ?

Another question, why TCP ?

Thanks
-Deepak
----- Original Message -----
From: "Brian Long" <brilong@cisco.com>
To: "Deepak Soneji" <sonejideepak@hotmail.com>
Cc: <toasters@mathworks.com>
Sent: Thursday, September 19, 2002 6:47 PM
Subject: Re: Sun Solaris 8 optimal performance with F880

> UDP and 32K transfers don't mix. Reduce to 8K transfers (nfs.udp.xfersize
> 8192) or change to TCP, the preferred method in which to use NFS.
>
> /Brian/
>
> > Toasters
> >
> > Configuration : Sun E6000 server running Solaris 8 latest patch level
> > Filer : F880
> > Cisco Switch 6509(Flowcontrol turned on)
> >
> >
> > We have recently bought 2- F880 filers and we are in the implementation
phases. The weird
> > performance results we are getting with following system settings and
mount parameters.
> > The output from "sysstat 1" on filer, shows writes 35 MBPS but reads
just very slow
> > or not even generate any traffic. Here are the settings
> >
> > ndd -set /dev/udp udp_recv_hiwat 65535
> > ndd -set /dev/udp udp_xmit_hiwat 65535
> > ndd -set /dev/tcp tcp_recv_hiwat 65535
> > ndd -set /dev/tcp tcp_xmit_hiwat 65535
> > ndd -set /dev/ge instance 1
> > ndd -set /dev/ge adv_pauseTX 1
> > ndd -set /dev/ge adv_pauseRX 1
> > ndd -set /dev/ge adv_1000autoneg_cap 0
> > ndd -set /dev/ge adv_1000fdx_cap 1
> >
> > set sq_max_size = 30
> > set nstrpush = 90
> > set ncsize 8000
> > set nfs:nfs3_max_threads = 24
> > set nfs:nfs3_nra =10
> > set nfs:nfs_max_threads = 24
> > set nfs:nfs_nra = 10
> >
> > Mount file system :
> >
> > mount -o rw,bg,hard,intr,proto=udp,vers=3,rsize=32768,wsize=32768
filer01:/vol/test /test
> >
> > Test method
> >
> > dd if=/dev/zero of=/test/foo bs=32k count=50000
> >
> >
> > I have tried to search some documents on now site for the optimal
performance but I could not
> > find any optimal numbers which I should try to achieve.
> > Why reads are chocking up ?
> > anyone has any proven configuration which works optimally ?
> >
> > Thanks in advance for your inputs and thoughts.
> >
> > Regards
> >
> > -Deepak Soneji
> > Flextronics International
> >
> >
>
> --
> Brian Long | | |
> Americas IT Hosting Sys Admin | .|||. .|||.
> Phone: (919) 392-7363 | ..:|||||||:...:|||||||:..
> Pager: (888) 651-2015 | C i s c o S y s t e m s
>

Re: Sun Solaris 8 optimal performance with F880 [ In reply to ]

Michael.Salmon at uab

Sep 23, 2002, 1:59 AM

Post #5 of 9 (3045 views)

Permalink

On Thursday, September 19, 2002 11:04:35 PM -0700 Deepak Soneji
<sonejideepak@hotmail.com> wrote:
+------
| Brian/Tim
|
| Great ! READS are greatly improved upto 12-15 mbps after changing xfersize
| to 8k.
| Any idea about what magic numbers for read/write I should try to achieve
| while tuning ?
|
| Another question, why TCP ?
+-----X8

UDP has no retry mechanism so if you loose any of the 22 packets involved
in a 32k transfer then you need to start again. A powerful server sending
to a weaker client is a good way to ensure that packets are lost. TCP has
mechanism to ensure that packets are not lost and hence works better when
the packet recipient is the weaker partner.

/Michael
--
This space intentionally left non-blank.

RE: Sun Solaris 8 optimal performance with F880 [ In reply to ]

mike.sphar at Remedy

Sep 23, 2002, 12:30 PM

Post #6 of 9 (3041 views)

Permalink

There is of course some additional overhead associated with the TCP protocol
though, which is why it really depends on the reliability of your network
and clients to determine which of the two protocols is ideal for your
environment.

Certainly in the situation being described, where a client is dropping
packets, TCP will likely be the better choice. (Though I'd also want to try
to address the dropped packet problem as well if possible.)

--
Mike Sphar - Sr Systems Administrator - Remedy Corporation

-----Original Message-----
From: Michael Salmon [mailto:Michael.Salmon@uab.ericsson.se]
Sent: Monday, September 23, 2002 3:59 AM
To: toasters@mathworks.com
Subject: Re: Sun Solaris 8 optimal performance with F880

On Thursday, September 19, 2002 11:04:35 PM -0700 Deepak Soneji
<sonejideepak@hotmail.com> wrote:
+------
| Brian/Tim
|
| Great ! READS are greatly improved upto 12-15 mbps after changing xfersize
| to 8k.
| Any idea about what magic numbers for read/write I should try to achieve
| while tuning ?
|
| Another question, why TCP ?
+-----X8

UDP has no retry mechanism so if you loose any of the 22 packets involved
in a 32k transfer then you need to start again. A powerful server sending
to a weaker client is a good way to ensure that packets are lost. TCP has
mechanism to ensure that packets are not lost and hence works better when
the packet recipient is the weaker partner.

/Michael
--
This space intentionally left non-blank.

Re: Sun Solaris 8 optimal performance with F880 [ In reply to ]

sonejideepak at hotmail

Sep 23, 2002, 8:07 PM

Post #7 of 9 (3046 views)

Permalink

I agree with you. Mostly I am going to use TCP and as you said
about dropped packets, we are working with Cisco to identify
the problem. At this moment, I am not confident of using UDP.
We will have several Oracle databases running on the filer and
I dont want to take any chances to corrupt the data.

Thanks
/Deepak

----- Original Message -----
From: "Mike Sphar" <mike.sphar@Remedy.COM>
To: <toasters@mathworks.com>
Sent: Monday, September 23, 2002 12:30 PM
Subject: RE: Sun Solaris 8 optimal performance with F880

> There is of course some additional overhead associated with the TCP
protocol
> though, which is why it really depends on the reliability of your network
> and clients to determine which of the two protocols is ideal for your
> environment.
>
> Certainly in the situation being described, where a client is dropping
> packets, TCP will likely be the better choice. (Though I'd also want to
try
> to address the dropped packet problem as well if possible.)
>
> --
> Mike Sphar - Sr Systems Administrator - Remedy Corporation
>
> -----Original Message-----
> From: Michael Salmon [mailto:Michael.Salmon@uab.ericsson.se]
> Sent: Monday, September 23, 2002 3:59 AM
> To: toasters@mathworks.com
> Subject: Re: Sun Solaris 8 optimal performance with F880
>
>
>
> On Thursday, September 19, 2002 11:04:35 PM -0700 Deepak Soneji
> <sonejideepak@hotmail.com> wrote:
> +------
> | Brian/Tim
> |
> | Great ! READS are greatly improved upto 12-15 mbps after changing
xfersize
> | to 8k.
> | Any idea about what magic numbers for read/write I should try to achieve
> | while tuning ?
> |
> | Another question, why TCP ?
> +-----X8
>
> UDP has no retry mechanism so if you loose any of the 22 packets involved
> in a 32k transfer then you need to start again. A powerful server sending
> to a weaker client is a good way to ensure that packets are lost. TCP has
> mechanism to ensure that packets are not lost and hence works better when
> the packet recipient is the weaker partner.
>
> /Michael
> --
> This space intentionally left non-blank.
>

RE: Sun Solaris 8 optimal performance with F880 [ In reply to ]

art at arzoon

Sep 24, 2002, 8:24 AM

Post #8 of 9 (3046 views)

Permalink

Deepak -- As I haven't been able to come up with any performance numbers for
the
700, 800, 900 series, can you let us know how your oracle db
performs on
the 880?

We are currently using an F760 with a gig network, tcp, etc and
we are
seeing a lot of I/O waits pertaining to Oracle temp and redos.
I'd
be interested in if you also see a bottleneck in that area.

Thanks

art hebert

-----Original Message-----
From: Deepak Soneji [mailto:sonejideepak@hotmail.com]
Sent: Monday, September 23, 2002 8:08 PM
To: Mike Sphar; toasters@mathworks.com
Subject: Re: Sun Solaris 8 optimal performance with F880

I agree with you. Mostly I am going to use TCP and as you said
about dropped packets, we are working with Cisco to identify
the problem. At this moment, I am not confident of using UDP.
We will have several Oracle databases running on the filer and
I dont want to take any chances to corrupt the data.

Thanks
/Deepak

----- Original Message -----
From: "Mike Sphar" <mike.sphar@Remedy.COM>
To: <toasters@mathworks.com>
Sent: Monday, September 23, 2002 12:30 PM
Subject: RE: Sun Solaris 8 optimal performance with F880

> There is of course some additional overhead associated with the TCP
protocol
> though, which is why it really depends on the reliability of your network
> and clients to determine which of the two protocols is ideal for your
> environment.
>
> Certainly in the situation being described, where a client is dropping
> packets, TCP will likely be the better choice. (Though I'd also want to
try
> to address the dropped packet problem as well if possible.)
>
> --
> Mike Sphar - Sr Systems Administrator - Remedy Corporation
>
> -----Original Message-----
> From: Michael Salmon [mailto:Michael.Salmon@uab.ericsson.se]
> Sent: Monday, September 23, 2002 3:59 AM
> To: toasters@mathworks.com
> Subject: Re: Sun Solaris 8 optimal performance with F880
>
>
>
> On Thursday, September 19, 2002 11:04:35 PM -0700 Deepak Soneji
> <sonejideepak@hotmail.com> wrote:
> +------
> | Brian/Tim
> |
> | Great ! READS are greatly improved upto 12-15 mbps after changing
xfersize
> | to 8k.
> | Any idea about what magic numbers for read/write I should try to achieve
> | while tuning ?
> |
> | Another question, why TCP ?
> +-----X8
>
> UDP has no retry mechanism so if you loose any of the 22 packets involved
> in a 32k transfer then you need to start again. A powerful server sending
> to a weaker client is a good way to ensure that packets are lost. TCP has
> mechanism to ensure that packets are not lost and hence works better when
> the packet recipient is the weaker partner.
>
> /Michael
> --
> This space intentionally left non-blank.
>

RE: Sun Solaris 8 optimal performance with F880 [ In reply to ]

Aleksandr.Rainchik at med

Sep 26, 2002, 3:20 PM

Post #9 of 9 (3050 views)

Permalink

As far as I know, that 32k/UDP block size problem is specific
to Cisco switches only. We had such a problem when our
swithces were upgraded from 55xx model to 65xx. Huge
performance problems with 32k blocks, had to change it to 8k.

Try to use crossover cable and connect same client directly
to the filer. You'll be surprised how well NFS with 32k/UDP
packets works...

-- a.r.

-----Original Message-----
From: Brian Long [mailto:brilong@cisco.com]
Sent: Thursday, September 19, 2002 8:48 PM
To: Deepak Soneji
Cc: toasters@mathworks.com
Subject: Re: Sun Solaris 8 optimal performance with F880

UDP and 32K transfers don't mix. Reduce to 8K transfers
(nfs.udp.xfersize
8192) or change to TCP, the preferred method in which to use NFS.

/Brian/

> Toasters
>
> Configuration : Sun E6000 server running Solaris 8 latest patch level
> Filer : F880
> Cisco Switch 6509(Flowcontrol turned on)
>
>
> We have recently bought 2- F880 filers and we are in the
implementation phases. The weird
> performance results we are getting with following system settings and
mount parameters.
> The output from "sysstat 1" on filer, shows writes 35 MBPS but reads
just very slow
> or not even generate any traffic. Here are the settings
>
> ndd -set /dev/udp udp_recv_hiwat 65535
> ndd -set /dev/udp udp_xmit_hiwat 65535
> ndd -set /dev/tcp tcp_recv_hiwat 65535
> ndd -set /dev/tcp tcp_xmit_hiwat 65535
> ndd -set /dev/ge instance 1
> ndd -set /dev/ge adv_pauseTX 1
> ndd -set /dev/ge adv_pauseRX 1
> ndd -set /dev/ge adv_1000autoneg_cap 0
> ndd -set /dev/ge adv_1000fdx_cap 1
>
> set sq_max_size = 30
> set nstrpush = 90
> set ncsize 8000
> set nfs:nfs3_max_threads = 24
> set nfs:nfs3_nra =10
> set nfs:nfs_max_threads = 24
> set nfs:nfs_nra = 10
>
> Mount file system :
>
> mount -o rw,bg,hard,intr,proto=udp,vers=3,rsize=32768,wsize=32768
filer01:/vol/test /test
>
> Test method
>
> dd if=/dev/zero of=/test/foo bs=32k count=50000
>
>
> I have tried to search some documents on now site for the optimal
performance but I could not
> find any optimal numbers which I should try to achieve.
> Why reads are chocking up ?
> anyone has any proven configuration which works optimally ?
>
> Thanks in advance for your inputs and thoughts.
>
> Regards
>
> -Deepak Soneji
> Flextronics International
>
>

--
Brian Long | | |
Americas IT Hosting Sys Admin | .|||. .|||.
Phone: (919) 392-7363 | ..:|||||||:...:|||||||:..
Pager: (888) 651-2015 | C i s c o S y s t e m s