Mailing List Archive

Pawel,
Are your file systems becoming full (type "df" from netapp")?
Mike


-----Original Message-----
From: Pawel Rogocz [mailto:pawel@rogocz.com]
Sent: Thursday, October 31, 2002 1:24 PM
To: toasters@mathworks.com
Subject: My F820 is slow ....


I run 6.2.1 on F820 and I have bunch of Sun boxes ( around 60 ) who do
NFS mounts.

The filer has more than one volume and sometimes during the day the output
of systat -x shows 100% disk util for a couple of minutes.

1. Is there a way to find out which volume/disk is causing bottleneck ?
2. Is there a way to find which client is causing the load to spike ?
3. Can disk util be monitored via SNMP ? I was not able to find anything
in the MIB provided on NOW.


thanks,


Pawel
--

On Thu, Oct 31, 2002 at 02:22:25PM -0600, Mike Ball wrote:
> Pawel,
> Are your file systems becoming full (type "df" from netapp")?
> Mike

Mine are at 50% in terms of disk space used.

1 volume, 12 disks, 18GB, F760, gige to client systems.

I show 91-94% cache hit rate, but disk utilization is 98-100% during
the business day.

--
Mike Horwath Admin & Manager @ VISI.com WORK: drechsau@visi.com
IRC: Drechsau http://www.visi.com/ HOME: drechsau@geeks.org
The only Minnesota ISP with public statistics: http://noc.visi.com/
Garbage In -- Gospel Out. - berkeley fortune(6)

Here it is:

/vol/vol1/ 338309500 292869428 45440072 87% /vol/vol1/
/vol/vol1/.snapshot 37589944 16548736 21041208 44%
/vol/vol1/.snapshot
/vol/vol2/ 338309500 303317696 34991804 90% /vol/vol2/
/vol/vol2/.snapshot 37589944 13034816 24555128 35%
/vol/vol2/.snapshot
/vol/vol0/ 50119928 1674084 48445844 3% /vol/vol0/
/vol/vol0/.snapshot 12529980 142548 12387432 1%
/vol/vol0/.snapshot
/vol/vol3/ 112769836 63064324 49705512 56% /vol/vol3/
/vol/vol3/.snapshot 12529980 20120056 0 161%
/vol/vol3/.snapshot
/vol/vol4/ 338309500 269767512 68541988 80% /vol/vol4/
/vol/vol4/.snapshot 37589944 13586164 24003780 36%
/vol/vol4/.snapshot




-----Original Message-----
From: Mike Ball [mailto:MBall@DATALINK.com]
Sent: Thursday, October 31, 2002 12:22 PM
To: 'Pawel Rogocz'; toasters@mathworks.com
Subject: RE: My F820 is slow ....


Pawel,
Are your file systems becoming full (type "df" from netapp")?
Mike


-----Original Message-----
From: Pawel Rogocz [mailto:pawel@rogocz.com]
Sent: Thursday, October 31, 2002 1:24 PM
To: toasters@mathworks.com
Subject: My F820 is slow ....


I run 6.2.1 on F820 and I have bunch of Sun boxes ( around 60 ) who do
NFS mounts.

The filer has more than one volume and sometimes during the day the output
of systat -x shows 100% disk util for a couple of minutes.

1. Is there a way to find out which volume/disk is causing bottleneck ?
2. Is there a way to find which client is causing the load to spike ?
3. Can disk util be monitored via SNMP ? I was not able to find anything
in the MIB provided on NOW.


thanks,


Pawel
--

Pawel,
Have you grown your volumes lately? If so, how many disks did you
add to the volumes when you grew them? I have seen customers who have
volumes that are at 90%-95% and they add one or two disks to increase the
disk space for the volume. What this does is creates a hot spot where any
new writes going to those volumes only gets written to the 1 or two disks
which were added = bad performance because WAFL is not able to stripe across
all the disks in the volume.
Mike


-----Original Message-----
From: Pawel Rogocz [mailto:pawel@rogocz.com]
Sent: Thursday, October 31, 2002 4:16 PM
To: Mike Ball; toasters@mathworks.com
Subject: RE: My F820 is slow ....


Here it is:

/vol/vol1/ 338309500 292869428 45440072 87% /vol/vol1/
/vol/vol1/.snapshot 37589944 16548736 21041208 44%
/vol/vol1/.snapshot
/vol/vol2/ 338309500 303317696 34991804 90% /vol/vol2/
/vol/vol2/.snapshot 37589944 13034816 24555128 35%
/vol/vol2/.snapshot
/vol/vol0/ 50119928 1674084 48445844 3% /vol/vol0/
/vol/vol0/.snapshot 12529980 142548 12387432 1%
/vol/vol0/.snapshot
/vol/vol3/ 112769836 63064324 49705512 56% /vol/vol3/
/vol/vol3/.snapshot 12529980 20120056 0 161%
/vol/vol3/.snapshot
/vol/vol4/ 338309500 269767512 68541988 80% /vol/vol4/
/vol/vol4/.snapshot 37589944 13586164 24003780 36%
/vol/vol4/.snapshot




-----Original Message-----
From: Mike Ball [mailto:MBall@DATALINK.com]
Sent: Thursday, October 31, 2002 12:22 PM
To: 'Pawel Rogocz'; toasters@mathworks.com
Subject: RE: My F820 is slow ....


Pawel,
Are your file systems becoming full (type "df" from netapp")?
Mike


-----Original Message-----
From: Pawel Rogocz [mailto:pawel@rogocz.com]
Sent: Thursday, October 31, 2002 1:24 PM
To: toasters@mathworks.com
Subject: My F820 is slow ....


I run 6.2.1 on F820 and I have bunch of Sun boxes ( around 60 ) who do
NFS mounts.

The filer has more than one volume and sometimes during the day the output
of systat -x shows 100% disk util for a couple of minutes.

1. Is there a way to find out which volume/disk is causing bottleneck ?
2. Is there a way to find which client is causing the load to spike ?
3. Can disk util be monitored via SNMP ? I was not able to find anything
in the MIB provided on NOW.


thanks,


Pawel
--

You said you had more than one volume ?.
Were the disks all added at the same time ?. That is, you have not added a
disk to an existing volume?. Is the activity read or write activity or a
mix ?.
Can you post up a sysstat output ?.
Try using 'statit' which will indicate if a specific disk is being hammered.
If this is the case then it would mean that a disk was added to the volume
group after vol creation and normal stripping is not in effect.

-----Original Message-----
From: Mike Horwath [mailto:drechsau@visi.com]
Sent: Friday, 1 November 2002 10:11
To: Mike Ball
Cc: 'Pawel Rogocz'; toasters@mathworks.com
Subject: Re: My F820 is slow ....


On Thu, Oct 31, 2002 at 02:22:25PM -0600, Mike Ball wrote:
> Pawel,
> Are your file systems becoming full (type "df" from netapp")?
> Mike

Mine are at 50% in terms of disk space used.

1 volume, 12 disks, 18GB, F760, gige to client systems.

I show 91-94% cache hit rate, but disk utilization is 98-100% during
the business day.

--
Mike Horwath Admin & Manager @ VISI.com WORK: drechsau@visi.com
IRC: Drechsau http://www.visi.com/ HOME: drechsau@geeks.org
The only Minnesota ISP with public statistics: http://noc.visi.com/
Garbage In -- Gospel Out. - berkeley fortune(6)

================================================================
This e-mail message has been scanned and cleared by MailMarshal
at gen-i limited - http://www.gen-i.co.nz
================================================================

Pawel


> 3. Can disk util be monitored via SNMP ? I was not able to find anything
> in the MIB provided on NOW.

I do not use this tool on our filers, but you can install MRTG on a box.
The instructions can be found at
http://now.netapp.com/NOW/download/tools/filer-mrtg/. It "provides a
current and historical view (the last day, week, month, and year) of
statistics for network traffic, CPU load, disk usage, and NFS and CIFS
operations."

-- Clarence.


> thanks,
>
>
> Pawel

Hi Pawel,
We have an F880 (6.2.1) with about 10-20 Sun boxes connected to ours. We
are experiencing the same type of slowdown and sysstat readouts. Our
database volumes are running particularly slow. I am working with NetApp now
to resolve this. Try the following set of commands;

Go into advanced mode on the filer (telnet in)
Priv set advanced
*you will see a star showing advanced mode
filer*> statit -b
leave for a few minutes
filer*>statit -enc

an nfsstat -l shows a detailed usage of nfs on a per client basis
This will give a detailed summary

Hope it helps,

Aaron


-----Original Message-----
From: Pawel Rogocz [mailto:pawel@rogocz.com]
Sent: Friday, November 01, 2002 5:24 AM
To: toasters@mathworks.com
Subject: My F820 is slow ....

I run 6.2.1 on F820 and I have bunch of Sun boxes ( around 60 ) who do
NFS mounts.

The filer has more than one volume and sometimes during the day the output
of systat -x shows 100% disk util for a couple of minutes.

1. Is there a way to find out which volume/disk is causing bottleneck ?
2. Is there a way to find which client is causing the load to spike ?
3. Can disk util be monitored via SNMP ? I was not able to find anything
in the MIB provided on NOW.


thanks,


Pawel
--

How many disk shelves have you got?
How many FCAL's you got?
Running any database devices on the filer?
What size raid groups have you got? Especially for databases.
Running any gigabit interfaces on Sun boxes?
Was the system running fast for a long period then suddenly performance
dropped or has it been gradual degradation?
Does a netdiag -v report any unusual NFS or application errors?

I have many more questions as I am going through something similar right
now. Hopefully we can help each other.

Thanks,
Aaron

-----Original Message-----
From: Neil Stichbury [mailto:Neil.Stichbury@gen-i.co.nz]
Sent: Friday, November 01, 2002 8:57 AM
Cc: 'Pawel Rogocz'; toasters@mathworks.com
Subject: RE: My F820 is slow ....

You said you had more than one volume ?.
Were the disks all added at the same time ?. That is, you have not added a
disk to an existing volume?. Is the activity read or write activity or a
mix ?.
Can you post up a sysstat output ?.
Try using 'statit' which will indicate if a specific disk is being hammered.
If this is the case then it would mean that a disk was added to the volume
group after vol creation and normal stripping is not in effect.

-----Original Message-----
From: Mike Horwath [mailto:drechsau@visi.com]
Sent: Friday, 1 November 2002 10:11
To: Mike Ball
Cc: 'Pawel Rogocz'; toasters@mathworks.com
Subject: Re: My F820 is slow ....


On Thu, Oct 31, 2002 at 02:22:25PM -0600, Mike Ball wrote:
> Pawel,
> Are your file systems becoming full (type "df" from netapp")?
> Mike

Mine are at 50% in terms of disk space used.

1 volume, 12 disks, 18GB, F760, gige to client systems.

I show 91-94% cache hit rate, but disk utilization is 98-100% during
the business day.

--
Mike Horwath Admin & Manager @ VISI.com WORK: drechsau@visi.com
IRC: Drechsau http://www.visi.com/ HOME: drechsau@geeks.org
The only Minnesota ISP with public statistics: http://noc.visi.com/
Garbage In -- Gospel Out. - berkeley fortune(6)

================================================================
This e-mail message has been scanned and cleared by MailMarshal
at gen-i limited - http://www.gen-i.co.nz
================================================================

Hey Clarence,
Filer MRTG is great, we run it now even with Datafabric Manager running as
well. It isn't very good for performance analysis but great for monitoring
filesystem usage and particularly snapshots if you are doing any
snapmirroring.

Thanks,
Aaron

-----Original Message-----
From: Clarence Ngoh [mailto:cngoh@bigpond.net.au]
Sent: Friday, November 01, 2002 9:09 AM
To: Pawel Rogocz; toasters@mathworks.com
Subject: Re: My F820 is slow ....

Pawel


> 3. Can disk util be monitored via SNMP ? I was not able to find anything
> in the MIB provided on NOW.

I do not use this tool on our filers, but you can install MRTG on a box.
The instructions can be found at
http://now.netapp.com/NOW/download/tools/filer-mrtg/. It "provides a
current and historical view (the last day, week, month, and year) of
statistics for network traffic, CPU load, disk usage, and NFS and CIFS
operations."

-- Clarence.


> thanks,
>
>
> Pawel

On Fri, Nov 01, 2002 at 10:56:40AM +1300, Neil Stichbury wrote:
> You said you had more than one volume ?.

No, Pawel did...

For me...

Single volume, started with 6 disks, added 6 more.

--
Mike Horwath Admin & Manager @ VISI.com WORK: drechsau@visi.com
IRC: Drechsau http://www.visi.com/ HOME: drechsau@geeks.org
The only Minnesota ISP with public statistics: http://noc.visi.com/
Garbage In -- Gospel Out. - berkeley fortune(6)