Hi all,
I noticed a bit of an odd issue with maintaining `known_hosts` when the
target machine is behind a bastion using `ProxyJump` or `ProxyCommand`
with host key clashes.
Client for me right now is OpenSSH_9.3p1 on Gentoo Linux/AMD64. I'm a
member of a team, and most of us use Ubuntu (yes, I'm a rebel). Another
team who actually maintain this fleet often access the same machines via
Windows 10/11 boxes (not sure if they use native OpenSSH or WSL). I
rather suspect this issue actually is not platform-specific.
Target machines are using OpenSSH on Debian/ARMHF (the exact version
varies with the exact OS version) -- hardware is essentially
industrialised Raspberry Pis.
The bastions are typically OpenWRT-based (Teltonica) routers with
Dropbear SSHd.
We share a configuration tree via a git repository which contains `Host`
entries for each of the target machines and the intermediate bastion hosts.
The target machines are mostly using "private" address space in the
172.16.0.0/12 subnet (although some are using 172.40.0.0/16 addresses,
because some goose thought all 172.0.0.0/8 subnets were "private"). The
bastion hosts run `dhcpd` and in many cases, the target acquires its
address via DHCP (yes, super bad idea).
That means that in many cases, multiple _different_ OpenSSH servers,
have the "same" IPv4 local address. Since when using `ProxyJump`, this
address is recorded along side the host's public key in the user's
`known_hosts` file, you can imagine when one logs into one server via
one bastion, then tries to log in to a different server via a different
bastion if that second server has the same local IPv4 address.
The crux of this is that we cannot assume the local IPv4 address is
unique, since it's not (and in many cases, not even static).
In the case of `ProxyCommand`, the IPv4 address of the target may not
even be obvious to the SSH client process.
-- Possible solutions / work-arounds using existing OpenSSH client --
I looked around for a solution, I ruled out turning off
`StrictHostKeyChecking` (as seen on ServerFault) as a terrible idea
asking for a Man-In-The-Middle attack.
DNS might "solve" the problem, but is likely to be messy to implement
(bastion hosts are resource constrained, not sure if they do dynamic DNS
for their LAN clients).
I know I'll get push-back from the other team if I try to mandate unique
local IPv4 addresses. (This is the same team that unwittingly decided
to rely on DHCP static assignment to "do the right thing".)
Link-local IPv6 is a tempting prospect: I have used this to get into a
target node when its DHCP client has gone AWOL leaving IPv4
unconfigured, and being derived from the MAC address, *should* be
globally unique, but this assumes a dual-stack LAN. (And the
engineering team who look after these are likely to baulk at this. They
barely understand IPv4!)
Port forwarding will require a lot of manual piss-farting around on the
router's config webpage… and will likely break if the embedded DHCPd
decides to not assign the static IP the target machine was supposed to get.
In the `ssh_config` man page, I see there is a `KnownHostsCommand`
option, which could possibly be employed here, however since the files
are "shared" by multiple users, there's the issue of paths, since I'll
bet the `KnownHostsCommand` is relative to ${PWD} and not ~/.ssh/config
or any config file imported by it.
User or Global `known_hosts` won't work due to the format of the file
used (it assumes a unique endpoint IP address, which we know is not
unique). (I have `HashKnownHosts` turned off on this Gentoo machine, my
workplace laptop has it turned on due to Ubuntu's default. Not sure if
this hash takes into account `ProxyJump` paths or `ProxyCommand` options.)
-- Possible solutions that require OpenSSH client changes --
One way that might work would be to embed the effective
`ProxyJump`/`ProxyCommand` path in the "host name" stored in
`known_hosts` -- will look ugly as sin, but at least the client can
"uniquely" identify each server, and determine which key to use for
validation.
e.g. you might have in known_hosts
172.16.1.2{ProxyJump user@10.20.30.40,user2@192.168.123.45} <algo> <key>
172.16.1.3{ProxyCommand user@10.20.30.40:nc 192.168.234.56 22} <algo> <key>
the {} part encodes the path by which you reach the host.
Alternative might be an "ExpectHostKey" option that can be put in
~/.ssh/config or specified with "-o ExpectHostKey=…" that tells the SSH
client "ignore your known_hosts file, the host *will* be using this
key". So if you know the public key (e.g. you did a `ssh_keyscan`), you
can either:
put in .ssh/config:
Host mytarget
Hostname 172.16.1.2
ProxyJump user2@bastion2
ExpectHostKey ecdsa-sha2-nistp256 AAAA…=
Host bastion2
Hostname 192.168.123.45
ProxyJump user@bastion1
ExpectHostKey ecdsa-sha2-nistp256 AAAA…=
Host bastion1
Hostname 10.20.30.40
ExpectHostKey ecdsa-sha2-nistp256 AAAA…=
OR, you might specify it on the command line (assuming the bastions are
"known")
ssh -o ExpectHostKey="ecdsa-sha2-nistp256 AAAA…=" \
-J user@bastion user@target
Bonus with this latter approach is that in a config sharing environment
using a SCM (whether it be git, Subversion, CVS… whatever), assuming
that repository was protected and "trusted", it would enable all members
of a team to automatically "trust" the host key with minimal
infrastructure set-up.
Are any of the above ideas feasible? Did I miss an obvious solution to
this?
--
Stuart Longland (aka Redhatter, VK4MSL)
I haven't lost my mind...
...it's backed up on a tape somewhere.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
I noticed a bit of an odd issue with maintaining `known_hosts` when the
target machine is behind a bastion using `ProxyJump` or `ProxyCommand`
with host key clashes.
Client for me right now is OpenSSH_9.3p1 on Gentoo Linux/AMD64. I'm a
member of a team, and most of us use Ubuntu (yes, I'm a rebel). Another
team who actually maintain this fleet often access the same machines via
Windows 10/11 boxes (not sure if they use native OpenSSH or WSL). I
rather suspect this issue actually is not platform-specific.
Target machines are using OpenSSH on Debian/ARMHF (the exact version
varies with the exact OS version) -- hardware is essentially
industrialised Raspberry Pis.
The bastions are typically OpenWRT-based (Teltonica) routers with
Dropbear SSHd.
We share a configuration tree via a git repository which contains `Host`
entries for each of the target machines and the intermediate bastion hosts.
The target machines are mostly using "private" address space in the
172.16.0.0/12 subnet (although some are using 172.40.0.0/16 addresses,
because some goose thought all 172.0.0.0/8 subnets were "private"). The
bastion hosts run `dhcpd` and in many cases, the target acquires its
address via DHCP (yes, super bad idea).
That means that in many cases, multiple _different_ OpenSSH servers,
have the "same" IPv4 local address. Since when using `ProxyJump`, this
address is recorded along side the host's public key in the user's
`known_hosts` file, you can imagine when one logs into one server via
one bastion, then tries to log in to a different server via a different
bastion if that second server has the same local IPv4 address.
The crux of this is that we cannot assume the local IPv4 address is
unique, since it's not (and in many cases, not even static).
In the case of `ProxyCommand`, the IPv4 address of the target may not
even be obvious to the SSH client process.
-- Possible solutions / work-arounds using existing OpenSSH client --
I looked around for a solution, I ruled out turning off
`StrictHostKeyChecking` (as seen on ServerFault) as a terrible idea
asking for a Man-In-The-Middle attack.
DNS might "solve" the problem, but is likely to be messy to implement
(bastion hosts are resource constrained, not sure if they do dynamic DNS
for their LAN clients).
I know I'll get push-back from the other team if I try to mandate unique
local IPv4 addresses. (This is the same team that unwittingly decided
to rely on DHCP static assignment to "do the right thing".)
Link-local IPv6 is a tempting prospect: I have used this to get into a
target node when its DHCP client has gone AWOL leaving IPv4
unconfigured, and being derived from the MAC address, *should* be
globally unique, but this assumes a dual-stack LAN. (And the
engineering team who look after these are likely to baulk at this. They
barely understand IPv4!)
Port forwarding will require a lot of manual piss-farting around on the
router's config webpage… and will likely break if the embedded DHCPd
decides to not assign the static IP the target machine was supposed to get.
In the `ssh_config` man page, I see there is a `KnownHostsCommand`
option, which could possibly be employed here, however since the files
are "shared" by multiple users, there's the issue of paths, since I'll
bet the `KnownHostsCommand` is relative to ${PWD} and not ~/.ssh/config
or any config file imported by it.
User or Global `known_hosts` won't work due to the format of the file
used (it assumes a unique endpoint IP address, which we know is not
unique). (I have `HashKnownHosts` turned off on this Gentoo machine, my
workplace laptop has it turned on due to Ubuntu's default. Not sure if
this hash takes into account `ProxyJump` paths or `ProxyCommand` options.)
-- Possible solutions that require OpenSSH client changes --
One way that might work would be to embed the effective
`ProxyJump`/`ProxyCommand` path in the "host name" stored in
`known_hosts` -- will look ugly as sin, but at least the client can
"uniquely" identify each server, and determine which key to use for
validation.
e.g. you might have in known_hosts
172.16.1.2{ProxyJump user@10.20.30.40,user2@192.168.123.45} <algo> <key>
172.16.1.3{ProxyCommand user@10.20.30.40:nc 192.168.234.56 22} <algo> <key>
the {} part encodes the path by which you reach the host.
Alternative might be an "ExpectHostKey" option that can be put in
~/.ssh/config or specified with "-o ExpectHostKey=…" that tells the SSH
client "ignore your known_hosts file, the host *will* be using this
key". So if you know the public key (e.g. you did a `ssh_keyscan`), you
can either:
put in .ssh/config:
Host mytarget
Hostname 172.16.1.2
ProxyJump user2@bastion2
ExpectHostKey ecdsa-sha2-nistp256 AAAA…=
Host bastion2
Hostname 192.168.123.45
ProxyJump user@bastion1
ExpectHostKey ecdsa-sha2-nistp256 AAAA…=
Host bastion1
Hostname 10.20.30.40
ExpectHostKey ecdsa-sha2-nistp256 AAAA…=
OR, you might specify it on the command line (assuming the bastions are
"known")
ssh -o ExpectHostKey="ecdsa-sha2-nistp256 AAAA…=" \
-J user@bastion user@target
Bonus with this latter approach is that in a config sharing environment
using a SCM (whether it be git, Subversion, CVS… whatever), assuming
that repository was protected and "trusted", it would enable all members
of a team to automatically "trust" the host key with minimal
infrastructure set-up.
Are any of the above ideas feasible? Did I miss an obvious solution to
this?
--
Stuart Longland (aka Redhatter, VK4MSL)
I haven't lost my mind...
...it's backed up on a tape somewhere.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
