Update is published:
https://www.netapp.com/us/media/tr-4067.pdf Will be another revision after editorial does a deeper edit.
From: Toasters <toasters-bounces@teaparty.net> On Behalf Of Parisi, Justin
Sent: Tuesday, July 7, 2020 10:34 AM
To: Fenn, Michael <fennm@DEShawResearch.com>; Scott Eno <cse@hey.com>; tmac <tmacmd@gmail.com>
Cc: Toasters <toasters@teaparty.net>
Subject: RE:
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
65536 can cause issues with ONTAP NFSv3 though. We only allow 128 per TCP connection. If a single client starts to overwhelm that, other clients have to wait to get resources, which creates latency. That’s why we recommend clients get set to no more than 128 slots, especially in environments that use many clients pointing to the same storage.
TR-4067’s update will have info on that, but for now, TR-4571 covers it on page 57.
https://www.netapp.com/us/media/tr-4571.pdf The new mount option best practice section for TR-4067 will be as follows (keep in mind that it’s under editorial review, so there may be a few changes in the new doc – mostly typo/spelling related):
Mount options
NFS mount option recommendations depend solely on the workload and application being used. There is general advice for specific mount options, but the decision on which NFS options to use is dependent on the client OS administrators and application vendor recommendations. There is no “catch all” recommendation for mount options. The following sections covers only a subset of NFS mount options. For a complete list of supported NFS mount options, use “man nfs” on your NFS client.
Default mount options
Linux clients will set default mount options out of the box. These defaults depend on client OS version and NFS configuration files found on the clients. Default mount options are what get set during a mount operation where no options are specified with the -o flag.
In some cases, mount options are negotiated with the NFS server. Specifically, in newer Linux kernels, the rsize/wsize values and the NFS versions are based on what the NFS server is set to.
For example, if NFSv4.1 is enabled and no NFS version is specified in configuration files or with the mount command, then the client will use NFSv4.1 as it is the highest supported NFS version enabled.
The following shows output from a mount command that issued no specific options. The ONTAP NFS server was set to 1MB TCP max transfer size (-tcp-max-xfer-size) and has NFSv4.1 enabled.
# mount DEMO:/flexgroup_16 /flexgroup
# mount | grep flexgroup
DEMO:/flexgroup_16 on /flexgroup type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.x.x.x,local_lock=none,addr=10.x.x.y)
Wsize/rsize
The mount options wsize and rsize determine how much data is sent between the NFS client and server for each packet sent. This may help optimize performance for specific applications, but should be set as per application vendor best practices, as what is best for one application may not be best for other applications.
Newer NFS clients will autonegotiate the wsize and rsize values to what the -tcp-max-xfer-size value is set to on the ONTAP NFS server if the mount command does not explicitly set the values. ONTAP defaults -tcp-max-xfer-size to 64K and can be set to a maximum of 1MB.
Note: The general recommendation for -tcp-max-xfer-size is to increase that value in ONTAP to 262144 (256K) and then specify explicit mount options as the applications require it.
For examples of some performance testing with different workload types and different wsize/rsize values, see “Performance Examples for Different TCP Max Transfer Window Sizes.”
Actimeo and Nocto
NFSv3 manages shared file system consistency and application performance by utilizing cached file/directory data and cached file/directory attributes. It is a loose consistency, because the application doesn’t have to go to shared storage and fetch data every time to use it. That can have tremendous impact to application performance. The cached information has timers that set the period to trust the cache data and at timeout, a light weight, fast getattr/access call to revalidate the data till the next time out.
There are two mechanisms that manage this process:
· Close to open consistency assures getting the latest data for a file, regardless of cache. (cto)
· Attribute cache timers. (actimeo; default 3s for file, 30s for directory)
If the client has complete ownership of data, ie it is not shared, there is guaranteed consistency. You can reduce getattr/access operations to storage and speed up the application by turning off cto consistency (nocto as a mount option) and by turning up the timeouts for the attribute cache management (actimeo=600 as a mount option changes the timer to 10m versus the defaults mentioned above). In some testing, nocto reduces ~65-70% of the getattr/access calls and actimeo will reduce another ~20-25%.
There are other cases that can benefit from a similar set of mount options, even though there is not complete ownership by the clients. For applications that use grids of clients like EDA, web hosting and movie rendering and have relatively static data sets (like the tools/libraries in EDA, like the web content for the web hosting, like the textures for rendering), the typical behavior is the data set is largely cached on the clients (very few reads; no writes). There will be many getattr/access calls coming back to storage in those cases. These data sets are typically updated through either another client mounting the file systems and periodically pushing content updates or in some case SnapMirror relationships to multiple file systems for update.
In these cases, there is a known lag in picking up new content and the application still works with potentially out of date data. For those scenarios, nocto and actimeo can be used to control the time period where out of data date can be managed. For example, in EDA with tools and libraries and other static content, actimeo=600 works well because this data is typically updated infrequently. For small web hosting where clients need to see their data updates timelier as they are editing their sites, actimeo=10 might be acceptable. For large scale web sites where there is content pushed to multiple file systems, actimeo=60 might be more effective. As always, test with your individual environmets.
Using this mount options reduce the workload to storage significantly in these cases (for example, a recent EDA experience reduced IOPs to the tool volume from >150K to ~6K) and applications can run significantly faster because they can trust the data in memory, rather than needing to query the NFS storage. This also helps reduce overall CPU % and load on the ONTAP nodes.
Actimeo
The actimeo mount option controls the attribute cache timeout on NFS clients. The actimeo option covers the entire range of available attribute caches, including:
acregmin=n
The minimum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 3-second minimum.
acregmax=n
The maximum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 60-second maximum.
acdirmin=n
The minimum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 30-second minimum.
acdirmax=n
The maximum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 60-second maximum.
Attribute caching provides some relief on networks by reducing the number of metadata calls. This also helps reduce latency to some workloads, as these metadata operations can now occur locally on the client. Attribute caching generally will have no impact to the number of overall operations, unless all operations to the storage were metadata – specifically ACCESS calls.
For example, in our Customer Proof of Concept labs, actimeo was set to 10 minutes (600 seconds) and saw latency cut in half with an EDA workload generated by vdbench (from ~2.08ms to ~1.05ms).
Figure 13) Default actimeo latency - vdbench.
[cid:image004.png@01D657C4.51B43760]
Figure 14) Actimeo=600 latency - vdbench.
[cid:image005.png@01D657C4.51B43760]
The downside of setting the actimeo value too high is that attributes that change might not be reflected properly until the cache timeout occurs, which could result in unpredictable access issues.
Note: The recommendation for attribute caching is to leave the defaults, unless otherwise directed by the application vendor or if testing shows a vast improvement in performance.
Nocto
Nocto stands for “no close-to-open,” which means that a file can close before a write has completed to save time. What this means in NFS environments is that other clients that have that file open for reading won’t get consistent updates to the file. By default, nocto is not set on NFS mounts, which means that all files will wait to finish writes before allowing a close.
The nocto option is used primarily to increase raw performance. For example, in the same vdbench tests run in our Customer Proof of Concept Labs, the nocto mount option reduced latency by an additional .35ms to .7ms.
Figure 15) Actimeo=600,nocto latency - vdbench.
[cid:image006.png@01D657C4.51B43760]
Note: The recommendation for use of the nocto option is to use only with read-heavy/read-mostly workloads.
Clientaddr
By default, the clientaddr mount option gets set automatically using the NFS client IP address. However, in some cases, the option may need to be specified in NFS mounts.
Two scenarios where it may be necessary to specify clientaddr would be:
· Multi-NIC clients may need to specify this option to ensure NFS uses the desired IP address for connectivity.
· NFSv4.x clients may need to specify this option if two clients with the same hostname (but different IP addresses) try to access NFS exports in the same SVM. NFSv4.x will send a client ID to the NFS server based on the hostname. ONTAP will respond with “CLID_IN_USE” and prevent the 2nd client from mounting if it uses the same client ID. Specifying clientaddr can force the client to increment the client ID on subsequent mount attempts.
Note: In most cases, clientaddr does not need to be specified.
Nconnect
A new NFS option called nconnect is in its nascent stages for use with NFS mounts. Currently, nconnect is only available on newer SLES, Ubuntu and Fedora clients, but is targeted for other Linux kernels in future OS versions. The purpose of nconnect is to provide multiple transport connections per TCP connection/mount point on a client. This helps increase parallelism and performance for NFS mounts. Details about nconnect and how it can increase performance for NFS in Cloud Volumes ONTAP can be found in this blog post:
The Real Baseline Performance Story: NetApp Cloud Volumes Service for AWS<
https://cloud.netapp.com/blog/aws-netapp-baseline-performance-testing>
Note: The recommendation for using nconnect depends on client OS and application needs. Testing with this new option is highly recommended before deploying in production.
Hard/soft
hard or soft mount options specify whether the program using a file using NFS should stop and wait (hard) for the server to come back online if the NFS server is unavailable or if it should report an error (soft).
If hard is specified, processes directed to an NFS mount that is unavailable cannot be terminated unless the intr option is also specified.
If soft is specified, the timeo=<value> option can be specified, where <value> is the number of seconds before an error is reported.
Note: For business-critical NFS exports, NetApp recommends using hard mounts.
Intr/nointr
intr allows NFS processes to be interrupted when a mount is specified as a hard mount. This policy is deprecated in new clients such as RHEL 6.4 and is hardcoded to “nointr.” Kill -9 is the only way to interrupt a process in newer kernels.
For business-critical NFS exports, NetApp recommends using intr with hard mounts with NFS clients that support it.
From: Fenn, Michael <fennm@DEShawResearch.com<mailto:fennm@DEShawResearch.com>>
Sent: Tuesday, July 7, 2020 10:29 AM
To: Scott Eno <cse@hey.com<mailto:cse@hey.com>>; tmac <tmacmd@gmail.com<mailto:tmacmd@gmail.com>>; Parisi, Justin <Justin.Parisi@netapp.com<mailto:Justin.Parisi@netapp.com>>
Cc: Toasters <toasters@teaparty.net<mailto:toasters@teaparty.net>>
Subject: Re:
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On more recent kernels, these are now sysctl parameters. You can put the following in /etc/sysctl.conf (or a new .conf file in /etc/sysctl.d) and run sysctl -w (or reboot):
sunrpc.tcp_max_slot_table_entries = 128
sunrpc.tcp_slot_table_entries = 128
sunrpc.udp_slot_table_entries = 64
That being said, you might want to check your current settings. At least on CentOS 7, sunrpc.tcp_max_slot_table_entries defaults to 65536, which is greater than the NetApp tuning, and should be more than large enough to ensure good write performance.
Side note, hard has been the default for a while now, and intr has been ignored for over a decade, so you can skip those (though they're not harmful to provide).
Thanks,
Michael
From: Scott Eno <cse@hey.com<mailto:cse@hey.com>>
Date: Tuesday, July 7, 2020 at 8:46 AM
To: tmac <tmacmd@gmail.com<mailto:tmacmd@gmail.com>>, "Parisi, Justin" <justin.parisi@netapp.com<mailto:justin.parisi@netapp.com>>
Cc: Toasters <toasters@teaparty.net<mailto:toasters@teaparty.net>>
Subject: Re: Re:
NetApp docs talk about these settings:
udp_slot_table_entries=64
tcp_slot_table_entries=128
tcp_max_slot_table_entries=128
as being for RHEL 6.3 and above. The system I'm helping a user with is Ubuntu 18.04.3. Do these settings still apply (noting there's currently no sunrpc.co, or sunrpc.conf, under modprobe.d).
"Parisi, Justin" <justin.parisi@netapp.com<mailto:justin.parisi@netapp.com>> wrote:
Just an FYI a new update to that TR is coming soon.
Get Outlook for iOS<
https://aka.ms/o0ukef>
________________________________
From: Toasters <toasters-bounces@teaparty.net<mailto:toasters-bounces@teaparty.net>> on behalf of tmac <tmacmd@gmail.com<mailto:tmacmd@gmail.com>>
Sent: Monday, July 6, 2020 9:51:07 PM
To: Scott Eno <cse@hey.com<mailto:cse@hey.com>>
Cc: Toasters <toasters@teaparty.net<mailto:toasters@teaparty.net>>
Subject: Re:
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.
https://www.netapp.com/us/media/tr-4067.pdf Hard, intr, nfsvers=3, rsize=65536,wsize=65536,proto=tcp,mountproto=tcp
Dont forget to modify the client to increase/set tcp_max_slot_table_entries to 128.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeam<
https://twitter.com/NetAppATeam>
I Blog at TMACsRack<
https://tmacsrack.wordpress.com/>
On Mon, Jul 6, 2020 at 8:45 PM Scott Eno via Toasters <toasters@teaparty.net<mailto:toasters@teaparty.net>> wrote:
---------- Forwarded message ----------
From: Scott Eno <cse@hey.com<mailto:cse@hey.com>>
To: Toasters <toasters@teaparty.net<mailto:toasters@teaparty.net>>
Cc:
Bcc:
Date: Tue, 07 Jul 2020 00:42:56 +0000
Subject: fstab nfsv3 mount favorites?
Hi,
It’s been a number of years since I worked in a Linux/NFS shop, so while I search for my old fstab mount options for a one-off project, does anyone have their go-to fstab nfsv3 mount options handy? The exported volume is on an ONTAP 9.6 cluster if it matters.
---------- Forwarded message ----------
From: Scott Eno via Toasters <toasters@teaparty.net<mailto:toasters@teaparty.net>>
To: Toasters <toasters@teaparty.net<mailto:toasters@teaparty.net>>
Cc:
Bcc:
Date: Tue, 7 Jul 2020 01:43:01 +0100
Subject:
_______________________________________________
Toasters mailing list
Toasters@teaparty.net<mailto:Toasters@teaparty.net>
https://www.teaparty.net/mailman/listinfo/toasters
