Mailing List Archive

Does a known security issue allow ssh login via system accounts?
Hi,

If this is not the right place to ask this, please redirect me. Hopefully it
is a known vulnerability, due to out of date software. We had a server
running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
week. The intruder managed to achieve this:

Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from 5.161.47.185 port 37962 ssh2

This despite that /etc/passwd has:

backup:x:34:34:backup:/var/backups:/usr/sbin/nologin

And /etc/shadow has:

backup:*:16359:0:99999:7:::

Either the /usr/bin/nologin or the "*" in the second field of /etc/shadow
should have been enough to prevent "Accepted password for backup." The
/usr/sbin/nologin is the standard version for that Ubuntu generation, byte
identical.

Adding this to sshd_config was effective:

DenyUsers backup

If that's still not the default for system-level users like "backup", would
adding it be a reasonble feature request? Or is that on the distros to
define their default sshd_config settings?

The files in pam.d on the compromised system are standard. There's no public
key for "backup", and no ".ssh" folder in /var/backups. The intruder managed
to send out spam via the local postfix service, which is what made the
intrusion obvious. OSSEC (Wazuh) didn't spot anything. We've of course taken
the system offline. But we'd like to understand how that login by "backup"
was possible.

Thanks for any pointers. It's hard to google for this, due to "backup" being
such a generic term.

Whit


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Mon, Feb 28, 2022 at 12:48:31PM -0500, Whit Blauvelt wrote:
> Hi,
>
> If this is not the right place to ask this, please redirect me. Hopefully it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
> week. The intruder managed to achieve this:
>
> Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from 5.161.47.185 port 37962 ssh2
>
> This despite that /etc/passwd has:
>
> backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
>
> And /etc/shadow has:
>
> backup:*:16359:0:99999:7:::
>
> Either the /usr/bin/nologin or the "*" in the second field of /etc/shadow
> should have been enough to prevent "Accepted password for backup." The
> /usr/sbin/nologin is the standard version for that Ubuntu generation, byte
> identical.
>
> Adding this to sshd_config was effective:
>
> DenyUsers backup
> ...

Possibly OT, but have you verified the system's PAM modules/configuration as well? They may
have used the backup account post-exploitation if everything else checked out.

-c
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 2022-02-28T18:55, Whit Blauvelt <whit@transpect.com> wrote:
> Hi,
>
> If this is not the right place to ask this, please redirect me. Hopefully it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
> week. The intruder managed to achieve this:
>
> Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from 5.161.47.185 port 37962 ssh2
>
> This despite that /etc/passwd has:
>
> backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
>
> And /etc/shadow has:
>
> backup:*:16359:0:99999:7:::
>
> Either the /usr/bin/nologin or the "*" in the second field of /etc/shadow
> should have been enough to prevent "Accepted password for backup." The
> /usr/sbin/nologin is the standard version for that Ubuntu generation, byte
> identical.

There are a few more mechanisms that may allow password logins despite
the measures you named: When using PAM authentication (which I would
assume, since you are talking about Ubuntu) all the configuration in
/etc/pam.d regarding ssh logins, so /etc/pam.d/ssh and
/etc/pam.d/common* will be in effect. If e.g. some module there
specifies a different password and shell for backup, e.g. from LDAP,
NIS, ActiveDirectory or different passwd/shadow files, those might
apply. You said below that pam.d is "standard", but installed packages
might modify configuration there, so I'd suggest having another look
there.

Also, depending on your configuration, /usr/sbin/nologin may not be
sufficient to prevent port-forwarding without an open shell, thus
enabling a spammer to forward some SMTP port for sending spam without
authentication by "looking like" coming from an internal server. This
is also hard to see and distinguish from an attacker having a shell in
the logs.

> Adding this to sshd_config was effective:
>
> DenyUsers backup
>
> If that's still not the default for system-level users like "backup", would
> adding it be a reasonble feature request? Or is that on the distros to
> define their default sshd_config settings?

I'd consider it good practice to selectively 'AllowGroup ssh-login' or
something to that effect for any internet-exposed system. But common
distros are far more open than that. I don't think OpenSSH is the right
place for a policy change, since as mentioned before, there is a lot
going on in PAM that will affect what OpenSSH does here, without being
part of OpenSSH.


Ciao,

Alexander W?rstlein.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
Hi Alexander,

Much appreciate the quick response and advice. May I ask for just a bit
more? It would be useful to find out more precisely how the intrusion worked
to know where its method should be appropriately reported.

> There are a few more mechanisms that may allow password logins despite
> the measures you named: When using PAM authentication (which I would
> assume, since you are talking about Ubuntu) all the configuration in
> /etc/pam.d regarding ssh logins, so /etc/pam.d/ssh and
> /etc/pam.d/common* will be in effect. If e.g. some module there
> specifies a different password and shell for backup, e.g. from LDAP,
> NIS, ActiveDirectory or different passwd/shadow files, those might
> apply. You said below that pam.d is "standard", but installed packages
> might modify configuration there, so I'd suggest having another look
> there.

Checking the files in /etc/pam.d/ and /lib/x86_64-linux-gnu/security/, they
exactly match the same sets of files on another Ubuntu system of the same
vintage with no public exposure, both in files there and each file's
particulars. There is no LDAP, NIS or AD use by the system. The string
"backup" is not in any file in /etc/pam.d/; the two places it obviously
shows up in /lib/x86_64-linux-gnu/security/*.so files are in message
strings, not references to the userid.

> Also, depending on your configuration, /usr/sbin/nologin may not be
> sufficient to prevent port-forwarding without an open shell, thus
> enabling a spammer to forward some SMTP port for sending spam without
> authentication by "looking like" coming from an internal server. This
> is also hard to see and distinguish from an attacker having a shell in
> the logs.

Likely right about port forwarding. When I caught the intrusion in
action "backup" did have an open ssh connection; unsure if "backup" had a
shell. Postfix was seeing the email origin as local.

> > Adding this to sshd_config was effective:
> >
> > DenyUsers backup
> >
> > If that's still not the default for system-level users like "backup", would
> > adding it be a reasonble feature request? Or is that on the distros to
> > define their default sshd_config settings?
>
> I'd consider it good practice to selectively 'AllowGroup ssh-login' or
> something to that effect for any internet-exposed system. But common
> distros are far more open than that. I don't think OpenSSH is the right
> place for a policy change, since as mentioned before, there is a lot
> going on in PAM that will affect what OpenSSH does here, without being
> part of OpenSSH.

Is there a valid use case where standard systems users (sync, backup, etc.)
should have ssh access? If not, blocking by default should be prudent. On
the other hand, if there are valid, standard uses for ssh for such users,
what are normal ways to enable them? I should see if any of them were used
here.

Highjacking a server for spam is a pedestrian goal; this evidences something
the hands of the script kiddies which requires better defense. Neither OSSEC
(Wazuh) on the system, nor regular scans with Nessus and OpenVAS, gave
warning. It would be good to find a signature for it; a start would be
adding a watch in OSSEC for system-level userids showing up in auth.log, but
there's some additional trick enabling those to bypass standard security.

Thanks again,

Whit


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 2022-02-28T22:16, Whit Blauvelt <whit@transpect.com> wrote:
> Hi Alexander,
>
> Much appreciate the quick response and advice. May I ask for just a bit
> more? It would be useful to find out more precisely how the intrusion worked
> to know where its method should be appropriately reported.
>
> > There are a few more mechanisms that may allow password logins despite
> > the measures you named: When using PAM authentication (which I would
> > assume, since you are talking about Ubuntu) all the configuration in
> > /etc/pam.d regarding ssh logins, so /etc/pam.d/ssh and
> > /etc/pam.d/common* will be in effect. If e.g. some module there
> > specifies a different password and shell for backup, e.g. from LDAP,
> > NIS, ActiveDirectory or different passwd/shadow files, those might
> > apply. You said below that pam.d is "standard", but installed packages
> > might modify configuration there, so I'd suggest having another look
> > there.
>
> Checking the files in /etc/pam.d/ and /lib/x86_64-linux-gnu/security/, they
> exactly match the same sets of files on another Ubuntu system of the same
> vintage with no public exposure, both in files there and each file's
> particulars. There is no LDAP, NIS or AD use by the system. The string
> "backup" is not in any file in /etc/pam.d/; the two places it obviously
> shows up in /lib/x86_64-linux-gnu/security/*.so files are in message
> strings, not references to the userid.

The string 'backup' wouldn't have to necessarily appear in the config.
Also, the .so files should be irrelevant, those are binaries, not config.

To check, I'd look at: PAM-Config (/etc/pam.d/*). That that config is
the same as another machine doesn't mean that there isn't a problem in
there. It just might mean that an attacker didn't introduce the problem
(or the problem was introduced in all machines). Then I'd look at
/etc/nsswitch.conf to check for other passwd/group/shadow mechanisms
than 'file'. Then I'd look at the relevant configs for those mechanisms.
Then I'd compare "getent passwd" to /etc/passwd, same for group and
shadow. There might be sssd in use, check config and cache. And check
/etc/shells, this should only contain proper shells, not anything like
/usr/bin/nologin, /bin/false, ...

The unfortunate problem is that Linux AAA has become a messy stack of
things that might cause problems. In the extreme case Authentication
might e.g. call into:
SSH -> /etc/pam.d/ssh -> /etc/pam.d/common* -> pam_unix.so ->
/etc/nsswitch.conf -> sssd (maybe cached) -> LDAP -> Kerberos

And there can be side-branches, e.g. SSH will also check for SSH-Keys,
PAM may try multiple modules one after another and can have
conditionals, same for nsswitch (but not as complex), same for sssd (but
even more complex).

In the sshd_config, there might also be Match-blocks or other
configuration that makes certain users be treated specially.

It is hard to say what your exact problem is withouth having seen all of
the above. There are just too many things that can go wrong and a lot of
"traditional nuances" in the configs one has to watch out for.

> > > Adding this to sshd_config was effective:
> > >
> > > DenyUsers backup
> > >
> > > If that's still not the default for system-level users like "backup", would
> > > adding it be a reasonble feature request? Or is that on the distros to
> > > define their default sshd_config settings?
> >
> > I'd consider it good practice to selectively 'AllowGroup ssh-login' or
> > something to that effect for any internet-exposed system. But common
> > distros are far more open than that. I don't think OpenSSH is the right
> > place for a policy change, since as mentioned before, there is a lot
> > going on in PAM that will affect what OpenSSH does here, without being
> > part of OpenSSH.
>
> Is there a valid use case where standard systems users (sync, backup, etc.)
> should have ssh access? If not, blocking by default should be prudent. On
> the other hand, if there are valid, standard uses for ssh for such users,
> what are normal ways to enable them? I should see if any of them were used
> here.

I don't see any common use cases for system users to be accessible via
SSH directly. Usually one uses /bin/false as a shell to disable login
for those users. /usr/bin/nologin does the same, but with a message.
This is what most distros do. I've seen problems with a homegrown
version of /usr/bin/nologin that displayed a multipage message with
'press any key to continue' and timeouts, which caused the exact
portforwarding problem you might have.

Another approach would be AllowGroups in the sshd_config, or appropriate
PAM configuration (there are tons of possibilities here).


Ciao,

Alexander W?rstlein.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Tue, 1 Mar 2022 at 04:52, Whit Blauvelt <whit@transpect.com> wrote:
> If this is not the right place to ask this, please redirect me. Hopefully it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7

What options did you configure it with? In particular, did you enable PAM?

[...]
> Either the /usr/bin/nologin or the "*" in the second field of /etc/shadow
> should have been enough to prevent "Accepted password for backup."

If you enabled PAM then that's a function of the PAM stack and its config.

> The /usr/sbin/nologin is the standard version for that Ubuntu generation,
> byte identical.

Have you verified that the sshd has not been tampered with?

> Adding this to sshd_config was effective:
> DenyUsers backup
> If that's still not the default for system-level users like "backup", would
> adding it be a reasonble feature request? Or is that on the distros to
> define their default sshd_config settings?

That would be up to the distros.

> The files in pam.d on the compromised system are standard.

"standard" as in "vendor-supplied" or "as we normally set them"?

> There's no public
> key for "backup", and no ".ssh" folder in /var/backups. The intruder managed
> to send out spam via the local postfix service, which is what made the
> intrusion obvious. OSSEC (Wazuh) didn't spot anything. We've of course taken
> the system offline. But we'd like to understand how that login by "backup"
> was possible.

I'd be having a very close look at the PAM config. I've seen multiple
instances where a misconfigured PAM stack failed open and accepted
either an empty password or any password. One instance also ended up
being used for spam as you describe.

You can use pam-test-harness.c (https://www.dtucker.net/patches/) to
test your config.

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Mon, 28 Feb 2022, Whit Blauvelt wrote:

> Hi,
>
> If this is not the right place to ask this, please redirect me. Hopefully it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
> week. The intruder managed to achieve this:
>
> Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from 5.161.47.185 port 37962 ssh2
>
> This despite that /etc/passwd has:
>
> backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
>
> And /etc/shadow has:
>
> backup:*:16359:0:99999:7:::

We're not aware of any security problems in OpenSSH 8.6 that could yield
access to a locked account like this.

On Ubuntu, OpenSSH delegates to PAM for password authentication and
verification of the user's shell against /etc/shells. You can check
whether this actually enabled using 'sudo sshd -T | grep ^usepam'

It sounds like you have already verified that your PAM configuration was
not tampered with, so that removes one possibility. Reviewing the Ubuntu
PAM configurations and the patches they apply to sshd seem to be prudent
next steps.

It's possible that the above log message was not from the initial
compromise, but rather the attacker running their own sshd to act as a
more reliable foothold.

Hopefully you're able to preseve and perform forensics on the host.
If there is evidence of a bug in sshd, then please send it to the
security bug reporting contact: openssh@openssh.com

Thanks,
Damien Miller
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Tue, 1 Mar 2022, Damien Miller wrote:

> We're not aware of any security problems in OpenSSH 8.6 that could yield
> access to a locked account like this.

I'd just add that if an attacker did have a sshd 0-day, then burning it
only to send spam seems amazingly profligate...

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
Hi Whit,

Did you solve the riddle already? When not, maybe use another
approach? My favorite is:

"HAL: I would recommend that we put the unit back in operation
and let it fail." (http://miscel.wikidot.com/2001-transcript)

Could you enable the old state with the spurious login being
possible again (maybe blocking external SSH access to the machine
for a moment), attach to the SSH root process by calling something
like

strace -o x.dump -s1024 -vvv -f -tt -p [pid]

and then try yourself to perform a password login as backup
to the machine?

Then hook SSH on another machine in the similar way and login
there too. Then compare the two dumps. If you you do not easily
see the difference try focusing only on the output of the PID
running the login process. Maybe also create to text files
with only the syscall names from both attempts and "diff" them
to see when things start getting weird.

If the syscall sequence is identical, the mmaps might be interesting
to compare. With same software version, the PROT_x flags and
also the file offsets should be identical. Differences indicate
backdoored libraries, sometimes even unmasking kernel root kits
hiding that changes with normal command line ls/stat - what might
be unlikely on a host just used for spamming. For that kind of
analysis grep "mmap" and sed works best (get rid of ASLR entropy
with sed).

Kind regards,
hd
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Mon, 02/28/22, 2022 at 11:17:12PM +0100, Alexander Wuerstlein wrote:

> To check, I'd look at: PAM-Config (/etc/pam.d/*). That that config is
> the same as another machine doesn't mean that there isn't a problem in
> there.

Right. Just saying if the problem's there, it's endemic to Ubuntu 16, as
those files don't differ from the same on other Ubuntu 16 systems

> It just might mean that an attacker didn't introduce the problem
> (or the problem was introduced in all machines). Then I'd look at
> /etc/nsswitch.conf to check for other passwd/group/shadow mechanisms
> than 'file'.

The nsswitch.conf is consistent even with Ubuntu 20 systems, except that the
latter have added systemd on the passwd and group lines.

> Then I'd look at the relevant configs for those mechanisms.

All I've found are standard.

> Then I'd compare "getent passwd" to /etc/passwd, same for group and
> shadow.

They each produce dumps identical to the files. However the VM has been
rebooted since the compromise, so this doesn't rule out prior subverion
there.

> There might be sssd in use, check config and cache.

I'm not familiar with sssd. Certainly never configured it. There's no "sssd"
process running.

> And check etc/shells, this should only contain proper shells, not anything
> /like usr/bin/nologin, /bin/false, ...

Just the standard list there.

> ... In the sshd_config, there might also be Match-blocks or other
> configuration that makes certain users be treated specially.

The match-blocks in sshd_config are only of the restrictive kind for certain
groups, not such as to add any extra permissions.

> It is hard to say what your exact problem is withouth having seen all of
> the above. There are just too many things that can go wrong and a lot of
> "traditional nuances" in the configs one has to watch out for.

> > Is there a valid use case where standard systems users (sync, backup, etc.)
> > should have ssh access? If not, blocking by default should be prudent. On
> > the other hand, if there are valid, standard uses for ssh for such users,
> > what are normal ways to enable them? I should see if any of them were used
> > here.
>
> I don't see any common use cases for system users to be accessible via
> SSH directly. Usually one uses /bin/false as a shell to disable login
> for those users. /usr/bin/nologin does the same, but with a message.
> This is what most distros do. I've seen problems with a homegrown
> version of /usr/bin/nologin that displayed a multipage message with
> 'press any key to continue' and timeouts, which caused the exact
> portforwarding problem you might have.

It's the standard Ubuntu 16 version of that file.

> Another approach would be AllowGroups in the sshd_config, or appropriate
> PAM configuration (there are tons of possibilities here).

What _is_ different here -- the only thing distinguishing this system from
stock Ubuntu 16.04.7 -- is the custom compile of a later (but now far from
latest) version of OpenSSH. The vulnerabilty looks to be either in Ubuntu
16.04.7, now beyond EOL, or OpenSSH_8.6p1 using OpenSSL 1.0.2g, also now
antiquated -- or their combination. Adding a DenyUsers line in sshd_config
listing all the system user accounts works to block this intrusion, and will
be my standard practice now. I found some discussion from about 8 years ago
claiming older versions of OpenSSH checked for the "nologin" shell (at least
in some configurations) and themselves blocked accordingly, which if true
would be equivalent to what adding that DenyUsers now can do.

Much appreciate all your help. Most generous! I only wish I'd have found
some additional clue by now.

Best,
Whit
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Tue, 03/01/22, 2022 at 09:21:48AM +1100, Darren Tucker wrote:

> What options did you configure it with? In particular, did you enable PAM?

./configure --with-md5-passwords --with-pam --with-privsep-path=/var/lib/sshd/ --sysconfdir=/etc/ssh8.6

> Have you verified that the sshd has not been tampered with?

Yes

> "standard" as in "vendor-supplied" or "as we normally set them"?

vendor-supplied.

> I'd be having a very close look at the PAM config. I've seen multiple
> instances where a misconfigured PAM stack failed open and accepted
> either an empty password or any password. One instance also ended up
> being used for spam as you describe.

> You can use pam-test-harness.c (https://www.dtucker.net/patches/) to
> test your config.

Thanks Darren. I'll try your test harness. My chief concern is whether there
is/was something off in the standard Ubuntu PAM setup.

Best,
Whit

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
I would echo Alexander's comment from up-thread and recommend failing closed, not open[*], by using 'AllowUsers' with a group containing the only users who should be able to ssh. (Whether you include root in that group depends on your use case).

____________________

[*]: If another system account is added to the machine and you don't add that account to the DenyUsers line, the new system account may be vulnerable to the same attack.

--
jmk

> On Mar 1, 2022, at 09:19, Whit Blauvelt <whit@transpect.com> wrote:
>
> Adding a DenyUsers line in sshd_config
> listing all the system user accounts works to block this intrusion, and will
> be my standard practice now.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Tue, 03/01/22, 2022 at 09:45:04AM +1100, Damien Miller wrote:

> It sounds like you have already verified that your PAM configuration was
> not tampered with, so that removes one possibility. Reviewing the Ubuntu
> PAM configurations and the patches they apply to sshd seem to be prudent
> next steps.

Found the culprit: me. I was stupid enough to install and configure for
libpam-google-auth, given a company mandate to 2FA all connections with
admin access, where it wasn't in scope to add 2FA to all client accounts. If
there's existing documentation anywhere on how dangerous this is, it's not
in libpam-google-auth's own docs, nor in the recipes scattered across the
net.

I've found no way yet to tweak it to be safe that I can be sure of, short of
running a separate sshd on another port for it. Has there been consideration
of adding 2FA to OpenSSH that doesn't require enabling PAM? Public keys and
IP restrictions seem enough to me. Yet my corporate overlord is required by
their insurance firm to use 2FA. To satisfy that demand, I compromised
security with the badly documented libpam-google-auth -- as if a firm that
can't even secure their flagship browser should be trusted on security.

Stupid me,

Whit

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 07/03/2022 16:14, Whit Blauvelt wrote:
> Found the culprit: me. I was stupid enough to install and configure for
> libpam-google-auth, given a company mandate to 2FA all connections with
> admin access, where it wasn't in scope to add 2FA to all client accounts.

Could you explain a bit more what went wrong?

In order to configure 2FA (*), I do this:

PasswordAuthentication no
UsePAM yes
AuthenticationMethods publickey,keyboard-interactive:pam

The comma means sshd requires publickey *and* PAM authentication to
complete.  I can't see how this can make it any less secure than
publickey alone, which is why I'm wondering what went wrong in your case.

If you want to disable 2FA either for certain trusted users, or certain
trusted source IP addresses, you can use a Match block, e.g.

Match Address 192.168.0.0/16
AuthenticationMethods publickey

On the other hand, if you're trying to enable *password* authentication
with separate 2FA, then that's certainly much trickier to get right in
the PAM stack.  I would go for public key or certificate auth instead,
with 2FA on top.

Alternatively, think about doing public key auth using ecdsa-sk keys and
a U2F token - they are very cheap. You need a fairly modern openssh at
client and server side though.

Regards,

Brian.

(*) I'm using pam_yubico, but I believe the above should apply to any
2FA challenge/response using PAM.


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 2022/03/07 11:14, Whit Blauvelt wrote:
> On Tue, 03/01/22, 2022 at 09:45:04AM +1100, Damien Miller wrote:
>
> > It sounds like you have already verified that your PAM configuration was
> > not tampered with, so that removes one possibility. Reviewing the Ubuntu
> > PAM configurations and the patches they apply to sshd seem to be prudent
> > next steps.
>
> Found the culprit: me. I was stupid enough to install and configure for
> libpam-google-auth, given a company mandate to 2FA all connections with
> admin access, where it wasn't in scope to add 2FA to all client accounts. If
> there's existing documentation anywhere on how dangerous this is, it's not
> in libpam-google-auth's own docs, nor in the recipes scattered across the
> net.
>
> I've found no way yet to tweak it to be safe that I can be sure of, short of
> running a separate sshd on another port for it. Has there been consideration
> of adding 2FA to OpenSSH that doesn't require enabling PAM? Public keys and

Already possible:

AuthenticationMethods "publickey,password"

Depending on what you are allowed to use as a second factor and what
clients you use, the key allowed by "publickey" could be a key from a
traditional id_XXX file, or it could be a key handle for a U2F token.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 3/7/22 17:14, Whit Blauvelt wrote:
> Found the culprit: me. I was stupid enough to install and configure for
> libpam-google-auth, given a company mandate to 2FA all connections with
> admin access,

libpam-google-auth and other similar PAM modules require to store the
token's shared secrets on the server. If your system gets hacked and
shared secrets are stolen the attacker can generate an arbitrary amount
of valid OTP values. And if you use the same shared secrets on multiple
servers the security impact will be broad.


> Has there been consideration
> of adding 2FA to OpenSSH that doesn't require enabling PAM? Public keys and
> IP restrictions seem enough to me.

Use the new FIDO key type.

Or use short-term OpenSSH user certificates issued by a secured SSH-CA
which uses 2FA for user authc.

Ciao, Michael.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 07/03/2022 16:38, Michael Ströder wrote:
> libpam-google-auth and other similar PAM modules require to store the
> token's shared secrets on the server. If your system gets hacked and
> shared secrets are stolen the attacker can generate an arbitrary
> amount of valid OTP values. And if you use the same shared secrets on
> multiple servers the security impact will be broad.
>
> => Don't use that.

That's a nice thing about pam_yubico with real Yubikeys: they can be
validated against the Yubico cloud API, without any local secrets.

I have also experimentally got TOTP validation working against a
Hashicorp Vault server: https://github.com/candlerb/vault-totp-helper

(I would be interested in having extra eyes on this)

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
>That's a nice thing about pam_yubico with real Yubikeys: they can be validated against the Yubico cloud API, without any local secrets.

Just to make sure I understand you correctly - a cloud service determines whether some access to your server is to be granted?
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
> >That's a nice thing about pam_yubico with real Yubikeys:
> >they can be validated against the Yubico cloud API,
> >without any local secrets.
>
> Just to make sure I understand you correctly - a cloud
> service determines whether some access to your server
> is to be granted?

A cloud service *authenticates* the user. It's the job of *other* PAM modules and configuration to decide what to *authorize* this authenticated identity for, including login.
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 07/03/2022 17:40, Philipp Marek wrote:
>> That's a nice thing about pam_yubico with real Yubikeys: they can be validated against the Yubico cloud API, without any local secrets.
> Just to make sure I understand you correctly - a cloud service determines whether some access to your server is to be granted?

Yes: a cloud service run by security experts, almost certainly more
secure and better managed than I could build myself. And in any case,
it's a second factor on top of private/public keys that I do manage locally.

(They used to provide code for you to host your own validation server -
but I've just checked and found that's end-of-life)
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Mon, 7 Mar 2022, Brian Candler wrote:

> Yes: a cloud service run by security experts, almost certainly more secure and
> better managed than I could build myself. And in any case, it's a second
> factor on top of private/public keys that I do manage locally.

Hope you have an axe nearby then ????

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
 ?  HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 2022/03/07 17:54, Brian Candler wrote:
> On 07/03/2022 17:40, Philipp Marek wrote:
> > > That's a nice thing about pam_yubico with real Yubikeys: they can be validated against the Yubico cloud API, without any local secrets.
> > Just to make sure I understand you correctly - a cloud service determines whether some access to your server is to be granted?
>
> Yes: a cloud service run by security experts, almost certainly more secure
> and better managed than I could build myself. And in any case, it's a second
> factor on top of private/public keys that I do manage locally.
>
> (They used to provide code for you to host your own validation server - but
> I've just checked and found that's end-of-life)

and written in PHP...

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 3/7/22 18:29, Brian Candler wrote:
> On 07/03/2022 16:38, Michael Ströder wrote:
>> libpam-google-auth and other similar PAM modules require to store the
>> token's shared secrets on the server. If your system gets hacked and
>> shared secrets are stolen the attacker can generate an arbitrary
>> amount of valid OTP values. And if you use the same shared secrets on
>> multiple servers the security impact will be broad.
>>
>> => Don't use that.
>
> That's a nice thing about pam_yubico with real Yubikeys: they can be
> validated against the Yubico cloud API, without any local secrets.

1. I'd never want to use tokens pre-provisioned tokens with admins'
shared secrets stored in the cloud and reach out to it via Internet from
every internal server.
Shameless plug: I'm using my own OATH-LDAP with shared secrets stored
encrypted in OpenLDAP, but not for SSH (see 3.).

2. Consider availability issues: In case of urgent admin SSH access your
infrastructure might be (partially) broken and Yubico's cloud unreachable.

3. Furthermore any OTP mechanism is not really usable when using tools
like ansible or similar in mass deployments.

BTW: Using keys on hardware tokens including U2F/Fido is also too slow
if you initially connect to thousands of machines.

=> Use an SSH-CA which issues short-term user certs.

Ciao, Michael.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On 07/03/2022 18:02, Stuart Henderson wrote:
> and written in PHP...

Yes, another good reason to use their cloud service.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Does a known security issue allow ssh login via system accounts? [ In reply to ]
On Mon, 7 Mar 2022, Brian Candler wrote:

> On 07/03/2022 18:02, Stuart Henderson wrote:
> > and written in PHP...
>
> Yes, another good reason to use their cloud service.

You mean that’s not written in PHP, using basically the same
code, but with probably less review?

No fan of outsourcing,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

****************************************************
/?\ The UTF-8 Ribbon
? ? Campaign against Mit dem tarent-Newsletter nichts mehr verpassen:
 ?  HTML eMail! Also, https://www.tarent.de/newsletter
? ? header encryption!
****************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

1 2  View All