Mailing List Archive

Merging GSSAPI kex?
I was thinking about the recent problems with the GSSAPI kex patch,
and I wonder if it would be best to merge GSSAPI kex into OpenSSH
upstream. My (admittedly dated) understanding is that the patch is
of high quality, and that the concerns are instead about the GSSAPI
implementation in use. However, I believe this is a non-issue for
most environments where GSSAPI kex would be useful: if someone can
find an RCE in the GSSAPI implementation, there are bigger problems
(like compromised domain controllers).

To avoid increasing the attack surface when GSSAPI is not in use,
I recommend having it off by default at both build-time and run-time.
The OpenBSD version would of course ship with it disabled (no GSSAPI
implementation in base), though there might be a package that
ships with it enabled. Most Linux distributions would ship with
it included in the build, but not enabled via sshd.conf. In this
configuration, I would expect there to be no drawbacks other than a
slightly increased binary size. I also believe that the additional
attack surface would be little greater than GSSAPI authentication,
which OpenSSH already supports.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
Re: Merging GSSAPI kex? [ In reply to ]
On Thu, May 26, 2022 at 10:43 AM Demi Marie Obenour
<demiobenour@gmail.com> wrote:

> I was thinking about the recent problems with the GSSAPI kex patch,
> and I wonder if it would be best to merge GSSAPI kex into OpenSSH
> upstream. My (admittedly dated) understanding is that the patch is
> of high quality, and that the concerns are instead about the GSSAPI
> implementation in use. However, I believe this is a non-issue for
> most environments where GSSAPI kex would be useful: if someone can
> find an RCE in the GSSAPI implementation, there are bigger problems
> (like compromised domain controllers).
>
> To avoid increasing the attack surface when GSSAPI is not in use,
> I recommend having it off by default at both build-time and run-time.
> The OpenBSD version would of course ship with it disabled (no GSSAPI
> implementation in base), though there might be a package that
> ships with it enabled. Most Linux distributions would ship with
> it included in the build, but not enabled via sshd.conf. In this
> configuration, I would expect there to be no drawbacks other than a
> slightly increased binary size. I also believe that the additional
> attack surface would be little greater than GSSAPI authentication,
> which OpenSSH already supports.

I agree with this, and will make the following observations:

1. The GSSAPI kex patch has been well-vetted.

Red Hat, Fedora, and at least several other Linux distributions have
included the GSSAPI kex patch in their official OpenSSH packages for
nearly a decade. (Red Hat Enterprise Linux 7, released on 2014-06-10,
was the first RHEL release to include it.) In terms of both breadth
and duration, that is a significant attack surface. In that time, to
my knowledge, there has been no security flaw discovered in OpenSSH
that was either enabled or worsened by the GSSAPI kex patch.

2. The GSSAPI kex patch is increasingly useful.

Over the past decade, the rise of tools such as sssd have made it
increasingly easy to join Unix hosts to Active Directory and rely on
Microsoft Active Directory for users, groups, and authentication (1).
At our organization, we join all of our Linux hosts to AD, and rely
extensively on Kerberos authentication that is enabled by all Linux
hosts possessing a Kerberos keytab, including ssh GSSAPI kex
authentication.

My team of 4 people manages over 400 Linux hosts, all joined to AD. I
have no interest in attempting to wrangle that many host ssh public
keys, and with GSSAPI kex, we do not have to.

3. Not including the GSSAPI kex patch likely leads to worse overall
real-world security.

One of the biggest Achilles’ heels of public/private key cryptography
is the secure dissemination of public keys. To attempt to securely
collect and securely publish all hosts’ respective ssh public keys is
a daunting task, and at least my experience has been that most
organizations do not perform it well. (There are always stale keys,
there are always some missing keys, et. al.)

If a user connects to a new host for the first time, and that host’s
public key is not yet available, the overwhelming temptation is to
assume that there is no MitM attacker and simply accept the presented
public key. You can argue users *shouldn’t* do that, but the reality
is that they do, and we all know that they do.

Excluding the GSSAPI kex patch due to theoretical (and arguably
unrealized) concerns over its security fitness, when the effect of
excluding it is to increase the opportunities for users to just accept
new ssh public keys without vetting them, is akin to demanding that
the front door of a house be made of 6” galvanized steel instead of 4”
galvanized steel when the house’s back door is a screen door. It is
the overall security of the system that matters, and the weakest link
in that overall security is almost always user behavior. GSSAPI kex
avoids giving users opportunities to do the wrong thing, and that
practical effect trumps any theoretical security concerns.

4. GSSAPI keys are rotated more frequently than ssh private keys.

Due to the challenge of ssh host public key distribution, I suspect
most sites do not attempt to regularly rotate (change) host ssh keys.
I am sure you could find organizations who operate long-standing hosts
that have been using the same ssh private keys for literally years.

In contrast, the design of Kerberos (mutual authentication, key
versioning) permits changing Kerberos keys as frequently as the
Kerberos TGT lifetime (typically 24 hours). By default, unless
overridden, an AD-joined host that runs sssd changes its host key
every 30 days (the ad_maximum_machine_account_password_age in
sssd.conf(5)). This drastically reduces the amount of time that a
potential attacker has to attempt brute-force cracking of the key, or
to impersonate the server (if the attacker were able to obtain the key
from the host keytab file).

So, in summary…

It was perfectly reasonable for the OpenSSH developers to express
security concerns about the GSSAPI kex patch when Simon first proposed
it. But now we have ~10 years of data that tell us that (fortunately)
those fears were not realized. These data are compelling that
including GSSAPI kex in OpenSSH will not weaken its overall security
posture—especially if GSSAPI kex is not enabled by default.

Moreover, GSSAPI kex neatly bypasses the thorny issue of secure a
priori distribution of host public ssh keys, the failings of which
enable arguably the biggest security weakness of ssh: users’ tendency
to trust a new host public ssh key without vetting it. Closing that
practical security risk results in an overall more secure system.

Finally, thanks to modern tools like sssd, an increasing number of
organizations can leverage GSSAPI kex, because their Linux hosts are
joined to AD and have a keytab file.

So I will respectfully second Demi’s request to the OpenSSH
developers: please reconsider your position on integrating the
long-standing unofficial GSSAPI kex patch into the official OpenSSH
distribution. OpenSSH is one of the most enormously useful tools in
any system administrator’s toolbox—and we are all indebted to you for
your effort over the years in maintaining and enhancing it.
Integrating the GSSAPI kex patch would only make it more useful to
system administrators everywhere.

Thank you for your consideration.

(1) https://sssd.io/
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Merging GSSAPI kex? [ In reply to ]
On Thu, May 26, 2022 at 11:45 PM James Ralston <ralston@pobox.com> wrote:

> On Thu, May 26, 2022 at 10:43 AM Demi Marie Obenour
> <demiobenour@gmail.com> wrote:
>
> > I was thinking about the recent problems with the GSSAPI kex patch,
> > and I wonder if it would be best to merge GSSAPI kex into OpenSSH
> > upstream. My (admittedly dated) understanding is that the patch is
> > of high quality, and that the concerns are instead about the GSSAPI
> > implementation in use. However, I believe this is a non-issue for
> > most environments where GSSAPI kex would be useful: if someone can
> > find an RCE in the GSSAPI implementation, there are bigger problems
> > (like compromised domain controllers).
> >
> > To avoid increasing the attack surface when GSSAPI is not in use,
> > I recommend having it off by default at both build-time and run-time.
> > The OpenBSD version would of course ship with it disabled (no GSSAPI
> > implementation in base), though there might be a package that
> > ships with it enabled. Most Linux distributions would ship with
> > it included in the build, but not enabled via sshd.conf. In this
> > configuration, I would expect there to be no drawbacks other than a
> > slightly increased binary size. I also believe that the additional
> > attack surface would be little greater than GSSAPI authentication,
> > which OpenSSH already supports.
>
> I agree with this, and will make the following observations:
>
> 1. The GSSAPI kex patch has been well-vetted.
>
> Red Hat, Fedora, and at least several other Linux distributions have
> included the GSSAPI kex patch in their official OpenSSH packages for
> nearly a decade. (Red Hat Enterprise Linux 7, released on 2014-06-10,
> was the first RHEL release to include it.) In terms of both breadth
> and duration, that is a significant attack surface. In that time, to
> my knowledge, there has been no security flaw discovered in OpenSSH
> that was either enabled or worsened by the GSSAPI kex patch.
>
> 2. The GSSAPI kex patch is increasingly useful.
>
> Over the past decade, the rise of tools such as sssd have made it
> increasingly easy to join Unix hosts to Active Directory and rely on
> Microsoft Active Directory for users, groups, and authentication (1).
> At our organization, we join all of our Linux hosts to AD, and rely
> extensively on Kerberos authentication that is enabled by all Linux
> hosts possessing a Kerberos keytab, including ssh GSSAPI kex
> authentication.
>
> My team of 4 people manages over 400 Linux hosts, all joined to AD. I
> have no interest in attempting to wrangle that many host ssh public
> keys, and with GSSAPI kex, we do not have to.
>
> 3. Not including the GSSAPI kex patch likely leads to worse overall
> real-world security.
>
> One of the biggest Achilles’ heels of public/private key cryptography
> is the secure dissemination of public keys. To attempt to securely
> collect and securely publish all hosts’ respective ssh public keys is
> a daunting task, and at least my experience has been that most
> organizations do not perform it well. (There are always stale keys,
> there are always some missing keys, et. al.)
>
> If a user connects to a new host for the first time, and that host’s
> public key is not yet available, the overwhelming temptation is to
> assume that there is no MitM attacker and simply accept the presented
> public key. You can argue users *shouldn’t* do that, but the reality
> is that they do, and we all know that they do.
>
> Excluding the GSSAPI kex patch due to theoretical (and arguably
> unrealized) concerns over its security fitness, when the effect of
> excluding it is to increase the opportunities for users to just accept
> new ssh public keys without vetting them, is akin to demanding that
> the front door of a house be made of 6” galvanized steel instead of 4”
> galvanized steel when the house’s back door is a screen door. It is
> the overall security of the system that matters, and the weakest link
> in that overall security is almost always user behavior. GSSAPI kex
> avoids giving users opportunities to do the wrong thing, and that
> practical effect trumps any theoretical security concerns.
>
> 4. GSSAPI keys are rotated more frequently than ssh private keys.
>
> Due to the challenge of ssh host public key distribution, I suspect
> most sites do not attempt to regularly rotate (change) host ssh keys.
> I am sure you could find organizations who operate long-standing hosts
> that have been using the same ssh private keys for literally years.
>
> In contrast, the design of Kerberos (mutual authentication, key
> versioning) permits changing Kerberos keys as frequently as the
> Kerberos TGT lifetime (typically 24 hours). By default, unless
> overridden, an AD-joined host that runs sssd changes its host key
> every 30 days (the ad_maximum_machine_account_password_age in
> sssd.conf(5)). This drastically reduces the amount of time that a
> potential attacker has to attempt brute-force cracking of the key, or
> to impersonate the server (if the attacker were able to obtain the key
> from the host keytab file).
>
> So, in summary…
>
> It was perfectly reasonable for the OpenSSH developers to express
> security concerns about the GSSAPI kex patch when Simon first proposed
> it. But now we have ~10 years of data that tell us that (fortunately)
> those fears were not realized. These data are compelling that
> including GSSAPI kex in OpenSSH will not weaken its overall security
> posture—especially if GSSAPI kex is not enabled by default.
>
> Moreover, GSSAPI kex neatly bypasses the thorny issue of secure a
> priori distribution of host public ssh keys, the failings of which
> enable arguably the biggest security weakness of ssh: users’ tendency
> to trust a new host public ssh key without vetting it. Closing that
> practical security risk results in an overall more secure system.
>
> Finally, thanks to modern tools like sssd, an increasing number of
> organizations can leverage GSSAPI kex, because their Linux hosts are
> joined to AD and have a keytab file.
>
> So I will respectfully second Demi’s request to the OpenSSH
> developers: please reconsider your position on integrating the
> long-standing unofficial GSSAPI kex patch into the official OpenSSH
> distribution. OpenSSH is one of the most enormously useful tools in
> any system administrator’s toolbox—and we are all indebted to you for
> your effort over the years in maintaining and enhancing it.
> Integrating the GSSAPI kex patch would only make it more useful to
> system administrators everywhere.
>
> Thank you for your consideration.
>
> (1) https://sssd.io/
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

I strongly second this proposal and could invest some time if necessary as
the current maintainer of OpenSSH in Red Hat.
--
Dmitry Belyavskiy
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Merging GSSAPI kex? [ In reply to ]
It's one solution for key distribution, but surely not the only one
and possibly not the best one. Popular doesn't equal good.

James Ralston wrote:
> These data are compelling that including GSSAPI kex in OpenSSH will
> not weaken its overall security posture—especially if GSSAPI kex is
> not enabled by default.

Dunno about that. Empirical evidence can only ever show that there
was no problem in the past. I guess some serious security issue has
existed in some project ~10 years before getting fixed.

More code, more complexity, in one of the most sensitive code paths
is not great.

Maybe this is rarely a primary concern where AD is used. One could
certainly argue that it should be.


> Integrating the GSSAPI kex patch would only make it more useful to
> system administrators everywhere.

Only to systems administrators wanting to use the functionality.

For everyone else in the world, probably including OpenSSH maintainers,
it can only make life worse.


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Merging GSSAPI kex? [ In reply to ]
On Thu, 26 May 2022, Demi Marie Obenour wrote:

> I was thinking about the recent problems with the GSSAPI kex patch,
> and I wonder if it would be best to merge GSSAPI kex into OpenSSH
> upstream. My (admittedly dated) understanding is that the patch is
> of high quality, and that the concerns are instead about the GSSAPI
> implementation in use. However, I believe this is a non-issue for most
> environments where GSSAPI kex would be useful: if someone can find
> an RCE in the GSSAPI implementation, there are bigger problems (like
> compromised domain controllers).

The main problem is that (AFAIK) none of the maintainers suffciently
know kerberos/GSSAPI nor use it regularly. The last time I used it was
over 10 years ago and I don't even have a working test setup for it ATM.

Additional impediments are consierable scepticism about the safety
of the GSSAPI implementations (e.g. none of them include fuzz tests
or are enrolled in oss-fuzz), the fact that this is definitionally
highly sensitivew pre-auth attack surface and an upstream that removed
Kerberos/GSSAPI from its base OS some years ago,

Not speaking for any of the other developers, but I don't see any
appetite or path to getting it merged.

That's not to say that the situation couldn't be improved for those who
want it. IMO getting a github project with the patches applied that
is kept up to date and with a real set of regression tests would be a
significant improvement over the status quo.

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Merging GSSAPI kex? [ In reply to ]
On 5/30/22 00:03, Damien Miller wrote:
> On Thu, 26 May 2022, Demi Marie Obenour wrote:
>
>> I was thinking about the recent problems with the GSSAPI kex patch,
>> and I wonder if it would be best to merge GSSAPI kex into OpenSSH
>> upstream. My (admittedly dated) understanding is that the patch is
>> of high quality, and that the concerns are instead about the GSSAPI
>> implementation in use. However, I believe this is a non-issue for most
>> environments where GSSAPI kex would be useful: if someone can find
>> an RCE in the GSSAPI implementation, there are bigger problems (like
>> compromised domain controllers).
>
> The main problem is that (AFAIK) none of the maintainers suffciently
> know kerberos/GSSAPI nor use it regularly. The last time I used it was
> over 10 years ago and I don't even have a working test setup for it ATM.

How is GSSAPI authentication (still supported IIRC) handled?

> Additional impediments are consierable scepticism about the safety
> of the GSSAPI implementations (e.g. none of them include fuzz tests
> or are enrolled in oss-fuzz),

That is definitely valid. That said, in most organizations I can
think of that would be interested in using GSSAPI kex, the same attack
surface can be reached in other ways. This is trivially true in an
Active Directory environment, for example.

> the fact that this is definitionally
> highly sensitivew pre-auth attack surface

Would it be attack surface even when disabled? The default should
obviously be disabled.

> and an upstream that removed
> Kerberos/GSSAPI from its base OS some years ago,>
> Not speaking for any of the other developers, but I don't see any
> appetite or path to getting it merged.
>
> That's not to say that the situation couldn't be improved for those who
> want it. IMO getting a github project with the patches applied that
> is kept up to date and with a real set of regression tests would be a
> significant improvement over the status quo.

--
Sincerely,
Demi Marie Obenour (she/her/hers)
Re: Merging GSSAPI kex? [ In reply to ]
On Mon, 30 May 2022, Demi Marie Obenour wrote:

> On 5/30/22 00:03, Damien Miller wrote:
> > On Thu, 26 May 2022, Demi Marie Obenour wrote:
> >
> >> I was thinking about the recent problems with the GSSAPI kex patch,
> >> and I wonder if it would be best to merge GSSAPI kex into OpenSSH
> >> upstream. My (admittedly dated) understanding is that the patch is
> >> of high quality, and that the concerns are instead about the GSSAPI
> >> implementation in use. However, I believe this is a non-issue for most
> >> environments where GSSAPI kex would be useful: if someone can find
> >> an RCE in the GSSAPI implementation, there are bigger problems (like
> >> compromised domain controllers).
> >
> > The main problem is that (AFAIK) none of the maintainers suffciently
> > know kerberos/GSSAPI nor use it regularly. The last time I used it was
> > over 10 years ago and I don't even have a working test setup for it ATM.
>
> How is GSSAPI authentication (still supported IIRC) handled?

Poorly - there is no functional testing AFAIK, only builds.

> > Additional impediments are consierable scepticism about the safety
> > of the GSSAPI implementations (e.g. none of them include fuzz tests
> > or are enrolled in oss-fuzz),
>
> That is definitely valid. That said, in most organizations I can
> think of that would be interested in using GSSAPI kex, the same attack
> surface can be reached in other ways. This is trivially true in an
> Active Directory environment, for example.
>
> > the fact that this is definitionally
> > highly sensitivew pre-auth attack surface
>
> Would it be attack surface even when disabled? The default should
> obviously be disabled.

Sure, but the costs to the developers should a an exploit be found are
essentially the same regardless of whether it is enabled by default or
not - we're still on the hook for analysis (of a system that we're not
familiar with), triage, potentially-urgent release and communication.

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Merging GSSAPI kex? [ In reply to ]
On 6/2/22 23:22, Damien Miller wrote:
> On Mon, 30 May 2022, Demi Marie Obenour wrote:
>
>> On 5/30/22 00:03, Damien Miller wrote:
>>> On Thu, 26 May 2022, Demi Marie Obenour wrote:
>>>
>>>> I was thinking about the recent problems with the GSSAPI kex patch,
>>>> and I wonder if it would be best to merge GSSAPI kex into OpenSSH
>>>> upstream. My (admittedly dated) understanding is that the patch is
>>>> of high quality, and that the concerns are instead about the GSSAPI
>>>> implementation in use. However, I believe this is a non-issue for most
>>>> environments where GSSAPI kex would be useful: if someone can find
>>>> an RCE in the GSSAPI implementation, there are bigger problems (like
>>>> compromised domain controllers).
>>>
>>> The main problem is that (AFAIK) none of the maintainers suffciently
>>> know kerberos/GSSAPI nor use it regularly. The last time I used it was
>>> over 10 years ago and I don't even have a working test setup for it ATM.
>>
>> How is GSSAPI authentication (still supported IIRC) handled?
>
> Poorly - there is no functional testing AFAIK, only builds.

Ouch. That sucks.

>>> Additional impediments are consierable scepticism about the safety
>>> of the GSSAPI implementations (e.g. none of them include fuzz tests
>>> or are enrolled in oss-fuzz),
>>
>> That is definitely valid. That said, in most organizations I can
>> think of that would be interested in using GSSAPI kex, the same attack
>> surface can be reached in other ways. This is trivially true in an
>> Active Directory environment, for example.
>>
>>> the fact that this is definitionally
>>> highly sensitivew pre-auth attack surface
>>
>> Would it be attack surface even when disabled? The default should
>> obviously be disabled.
>
> Sure, but the costs to the developers should a an exploit be found are
> essentially the same regardless of whether it is enabled by default or
> not - we're still on the hook for analysis (of a system that we're not
> familiar with), triage, potentially-urgent release and communication.

Okay, that makes sense. I hadn’t considered the possibility of
an exploit against the code that interfaces between OpenSSH and the
GSSAPI. A vulnerability in the GSSAPI itself would obviously be the
responsibility of whoever maintains the GSSAPI implementation, not you.

One option (which might be a terrible one ????) would be to implement
GSSAPI kex support as a plugin. That would allow it to be maintained
by those who are actually familiar with the code. It would also
allow GSSAPI auth to be made part of the plugin, meaning that you
would not need to worry about it anymore.

--
Sincerely,
Demi Marie Obenour (she/her/hers)