Mailing List Archive

Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4
Greetings,
Just reaching out for experiences on Openstack deployments with Netapp
NFS. Are you using the default v4 or have switched the mount options to
v3? We have a pretty good base of ISCSI, but based on performance and
manageability are considering making the jump.

Speaking of NFSv4, outside of openstack has anyone made any large
deployments? Is it as solid as v3? using uid/gid, is anyone using sec=krb5?



Regards,
Douglas
Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4 [ In reply to ]
Crickets ? :)

Alright let us remove openstack qualifier. Does anyone use v4 or know
anyone that does with Netapp FAS?

On Tue, Sep 22, 2020 at 6:53 PM Douglas Siggins <siggins@gmail.com> wrote:

> Greetings,
> Just reaching out for experiences on Openstack deployments with Netapp
> NFS. Are you using the default v4 or have switched the mount options to
> v3? We have a pretty good base of ISCSI, but based on performance and
> manageability are considering making the jump.
>
> Speaking of NFSv4, outside of openstack has anyone made any large
> deployments? Is it as solid as v3? using uid/gid, is anyone using sec=krb5?
>
>
>
> Regards,
> Douglas
>
RE: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4 [ In reply to ]
I hope I’m doing this right… (posting to the list I mean).

I have four MySQL servers using NFSv4, but not with krb5. The rest of the environment is v3.

We did do a pilot with OpenStack Container Platform and we used v3 (v4 was enabled, so it may have used it) provisioning via Trident with success. We also used Trident to provision iSCSI volumes. Worked quite well.

Hope that helps a little.

Wayne

From: Toasters <toasters-bounces@teaparty.net> On Behalf Of Douglas Siggins
Sent: Friday, September 25, 2020 9:01 AM
To: Toasters <toasters@teaparty.net>
Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4


CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Crickets ? :)

Alright let us remove openstack qualifier. Does anyone use v4 or know anyone that does with Netapp FAS?

On Tue, Sep 22, 2020 at 6:53 PM Douglas Siggins <siggins@gmail.com<mailto:siggins@gmail.com>> wrote:
Greetings,
Just reaching out for experiences on Openstack deployments with Netapp NFS. Are you using the default v4 or have switched the mount options to v3? We have a pretty good base of ISCSI, but based on performance and manageability are considering making the jump.

Speaking of NFSv4, outside of openstack has anyone made any large deployments? Is it as solid as v3? using uid/gid, is anyone using sec=krb5?



Regards,
Douglas
RE: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4 [ In reply to ]
Re: v3 vs. v4.x security...

With NFSv3, the sideband protocols are not Kerberized when with krb5*. So all that gets kerberized is the NFS protocol; mount, portmap, nlm, etc are all plaintext. That's one aspect of security that v4.x provides with everything encapsulated. Those sideband protocols can contain some information about the mount that you don't want to be readable.

For example, MOUNT shows:

- machine account name
- uid/gid mounting
- auxiliary groups
- mount path
- filehandle info

NFSv4.x stores all that info with PUTROOTFH calls in the NFS packets, so that all gets GSS wrapped.

Plus, you can configure v4.x to force domain string ID auth only, whereas v3 is numeric ID only. That offers some flexibility (such as being able to have different UIDs on storage and client, provided they can both resolve name@domain.com to a user), but also offers some security benefits when you tack on Kerberos UPN.

V4.x does offer some security benefits that v3 does not, but you're right that you can accomplish most of the v4 security stuff in v3.

-----Original Message-----
From: Michael Bergman <michael.bergman@ericsson.com>
Sent: Monday, October 5, 2020 6:18 AM
To: Toasters <toasters@teaparty.net>
Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4

While the locking mechanism is understandable, the side band way (NLM
protocol) in v3 really is a bummer (very very bad performance for example) this surprises me:

"Their business makes it a requirement for the security"

There's really nothing w.r.t. "security" in v4.1 and up that is crucial or important that cannot, with the same effort (client side effort...), be implemented with v3. In practice. There's the sec=krb5{,i,p} stuff (old 7-mode syntax there I know). All is supported in all sensible Linux distros I'm aware of (and Oracle Solaris if anyone cares about that anymore I'm not sure). Other NFS client implementations, sure, YMMV. But it should be residual by now.

Another dimension of security is what Justin pointed out below about NVSv4
ACLs: ONTAP allows you to "nfsv4acls-nfsv3mounts".

Remains one more thing then (in addition to the first thing I wrote): the fact that NFSv4.x uses one port 2049. Everything. So it's easier to "open the Firewall".
But seriously. How hard can it be with NFSv3 and those few side band protocols? It's just a few more ports, in total like 5 (TCP and UDP for the side band protocols: rpcbind, MOUNT, NLM, status. Who uses rquota these
days...??) It's not rocket science. We have tons of FW openings for NFSv3 here in our complex network environments internally inside Ericsson's "Customer Zero" (our Engineering and Telco Test environments). It's rarely, if ever, a problem per se. The problem is that the storage traffic is very intense and tends to kill the Firewalls. There are so many sessions going on at the same time that if you put a FW in the middle of the (*one* Firewall mind you...) its CPUs will be at 100% sooner or later

Last but not least: yes there are fundamental performance issues (still) with v4.x. We have done investigations and it's not looking good vs NFSv3.
Yes, it's possible to get v4.1 (and pNFS) to perform well if you can get your workload to parallelise very much -- scale out.
Be prepared that the work to fiddle with everything and tune all the details to get it it to perform well for you in your particular environment is VERY cumbersome.

If not, if you have "vertical" workload threads that need to do *a lot* (incl metadata) on one (1) mount point, then high risk for a painful experience with ONTAP's v4.1 today. Bottlenecks which have to be removed first and it will take time

/M

-------- Original Message --------
Subject: RE: Re:
Date: Fri, 2 Oct 2020 13:18:57 +0000
From: Parisi, Justin <Justin.Parisi@netapp.com>
To: Sebastian P. Goetze <spgoetze@gmail.com>, Douglas Siggins
<siggins@gmail.com>
CC: Toasters <Toasters@teaparty.net>

We get lots of questions about NFSv4, especially for the security and
locking aspects.

The main issues people see with it are performance (especially with high
metadata workloads) and complexity in setup.

It's a stateful protocol, so it also sees disruption during failovers.

Mostly, people use it for the following reasons:

* Their business makes it a requirement for the security
* They have an application that requires it for the locking
* They want ACLs
* They only want to open up a single port over a firewall

If you just want it for ACLs, ONTAP allows you to set v4 ACLs and still use
NFSv3:

https://whyistheinternetbroken.wordpress.com/2017/08/11/nfsv4acls-nfsv3mounts/

_________________________________________________
*From:* Toasters <toasters-bounces@teaparty.net> *On Behalf Of *Sebastian P.
Goetze
*Sent:* Friday, October 2, 2020 3:26 AM
*To:* Douglas Siggins <siggins@gmail.com>
*Cc:* Toasters <Toasters@teaparty.net>
*Subject:* Re:

I teach NetApp and do have students once in a while using NFS v4, but
usually only when necessary, e.g. routing through firewalls (which in my
experience seems to be the #1 reason using v4). I did have people using pNFS
in production, but that's rare.

Some are evaluating, though. Maybe when multipathing is supported (regular
like VMware, not pNFS), there will be an uptick in adoption...

My 2c

Greetings from Northfriesland

Sebastian

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
https://www.teaparty.net/mailman/listinfo/toasters
RE: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4 [ In reply to ]
Root gets access to Kerberos depending on what you mapped the client SPN to in ONTAP. If you mapped it to root, it gets root. If you mapped it to another user, it becomes that user and gets that user's permissions.

-----Original Message-----
From: Michael Bergman <michael.bergman@ericsson.com>
Sent: Monday, October 5, 2020 9:43 AM
To: Parisi, Justin <Justin.Parisi@netapp.com>
Cc: Toasters <toasters@teaparty.net>
Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4

NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.




You're right. The krb5 is not covering the side band protocols pre- v4.x. I forgot about that bit, thanks for the reminder.

In some "top secret" environments, I could see that sniffing in the Network and seeing things like Computer Object name, uid/gid, mount path and FH could by someone be seen as a security risk. *Eve* if the actual datagrams with user data could never be seen. For most use cases, I contend that this is just an ISO 27001 tick box item, though.

A thought:
OTOH if anyone, anyone at all, is root on an NFS client then can they not somehow from inside it see that MOUNT information anyway? (Even if the info is encrypted in transit with krb5 then.) I'm not sure how that bit works, never looked into it

Justin Parisi wrote:
> you can configure v4.x to force domain string ID auth only [...]
> ,but also offers some security benefits when you tack on Kerberos UPN.

I did try a setup with this at some point in the past and then got really tired of the whole thing and tore it down. It's just... cumbersome. I can't think of any non esoteric Linux (Unix) based environment with lots of NFS traffic which would be reasonably manageable with such a config

/M

-------- Original Message --------
Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4
Date: Mon, 5 Oct 2020 13:29:19 +0000
From: Parisi, Justin <Justin.Parisi@netapp.com>
To: NGC-michael.bergman-ericsson.com <michael.bergman@ericsson.com>, Toasters <toasters@teaparty.net>

Re: v3 vs. v4.x security...

With NFSv3, the sideband protocols are not Kerberized when with krb5*. So all that gets kerberized is the NFS protocol; mount, portmap, nlm, etc are all plaintext. That's one aspect of security that v4.x provides with everything encapsulated. Those sideband protocols can contain some information about the mount that you don't want to be readable.

For example, MOUNT shows:

- machine account name
- uid/gid mounting
- auxiliary groups
- mount path
- filehandle info

NFSv4.x stores all that info with PUTROOTFH calls in the NFS packets, so that all gets GSS wrapped.

Plus, you can configure v4.x to force domain string ID auth only, whereas v3 is numeric ID only. That offers some flexibility (such as being able to have different UIDs on storage and client, provided they can both resolve name@domain.com to a user), but also offers some security benefits when you tack on Kerberos UPN.

V4.x does offer some security benefits that v3 does not, but you're right that you can accomplish most of the v4 security stuff in v3.

_______________________________________________
Toasters mailing list
Toasters@teaparty.net
https://www.teaparty.net/mailman/listinfo/toasters