Mailing List Archive

Nico Kadel-Garcia wrote:
> I'm dealing with Azure DevOps git services with which recent, security
> hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> based SSH links to the Azure Devops.

How was that security hardning done?

And if it's something RHEL-specific then what about upstream OpenSSH?


> Conversely, Azure Bastion cannot use Azure key vault stored private
> SSH keys to access the same RHEL 8 based servers,

Try running sshd with -ddd and see what it says?

Although this seems to be a RHEL issue maybe you can debug it yourself..


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Sat, May 21, 2022 at 9:38 AM Peter Stuge <peter@stuge.se> wrote:
>
> Nico Kadel-Garcia wrote:
> > I'm dealing with Azure DevOps git services with which recent, security
> > hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> > based SSH links to the Azure Devops.
>
> How was that security hardning done?

I rather wish I had every detail of the hardening: it's the CIS
published L2 security, RHEL 8 VM image in the Azure Marketplace. I
don't have in my personal hands the precise set of hardening steps. I
wish I did! I'm not a big believer that different vendors or security
staff follow precisely the same steps to follow a published security
standard, even if it's me doing the work, and it has been me on
numerous occasions in my career. It's why when I do those, I publish
the dockerfiles or chef or ansible playbooks applied to the VM image
as a published list, in source control for the employere.

In this case, the Azure VM with which I'm having the issues is listed
by the "az vm image list --all -otsv" command as:

center-for-internet-security-inc:cis-rhel-8-l2:cis-rhel8-l2:1.0.11


> And if it's something RHEL-specific then what about upstream OpenSSH?

It works fine on CentOS and standard RHEL based images, it seems
specific to the CIS published image. I was hoping someone else had
seen the issue.

For reference, I'm an *old* SSH users, and published the first public
ports of ssh-1, ssh-2, and OpenSSH to SunOS back in the day, and still
take potshots at backporting OpenSSH current releases to RHEL.

> > Conversely, Azure Bastion cannot use Azure key vault stored private
> > SSH keys to access the same RHEL 8 based servers,
>
> Try running sshd with -ddd and see what it says?
>
> Although this seems to be a RHEL issue maybe you can debug it yourself..

Since the dominating problem is the git client usage, I'm more
concerned with running:

GIT_SSH_COMMAND='git -v -v -v -v -v' git clone azure-server-git-url

I don't have manual control of the git server enough to tweak the
'sshd' options on it: it's a cloud service, and I suspect I'd need a
much, much larger contract and much more established credentials with
the "Tier 3" engineers on that project to get them to tweak the sshd
options.

It seems CIS specific, above and beyond being RHEL specific. I did
synchronize, and revert, the 'gnutls' configs set by the
'crypto-policies' RPMs, to avoid conflicts among the sets of
encryption protocols listed in

/etc/crypto-policies/back-ends/libssh.config
/etc/crypto-policies/back-ends/openssh.config
/etc/crypto-policies/back-ends/opensshserver.config

I speculated that a published tweak there among the
'PubkeyAcceptedKeyTypes' might be forbidding a required encryption
protocol. I was hoping someone else had already run into the issue and
had a working fix or pointer to specific workarounds. So far, no luck.

I was kind of hoping someone else with my kind of multi-environment
support experience had run into this and had a workaround. I'd also
like a pony....


>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 2022-05-21 at 14:04 -0400, Nico Kadel-Garcia wrote:
> Since the dominating problem is the git client usage, I'm more
> concerned with running:
>
> GIT_SSH_COMMAND='git -v -v -v -v -v' git clone azure-server-git-url

I think you mean
GIT_SSH_COMMAND='ssh -v -v -v -v -v' git clone azure-server-git-url

but it may be easier to just work with
ssh -v -v -v azure-server-git-server git-upload-pack /url


I'm afraid I have no pony for you, Nico. My guess is that the CIS
hardening removed something (e.g. ssh-rsa) which is the only one
supported by that Azure git service, but you surely had a similar guess
already.


Regards


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Sat, May 21, 2022 at 6:01 PM Ángel <angel@pgp.16bits.net> wrote:
>
> On 2022-05-21 at 14:04 -0400, Nico Kadel-Garcia wrote:
> > Since the dominating problem is the git client usage, I'm more
> > concerned with running:
> >
> > GIT_SSH_COMMAND='git -v -v -v -v -v' git clone azure-server-git-url
>
> I think you mean
> GIT_SSH_COMMAND='ssh -v -v -v -v -v' git clone azure-server-git-url

Quite right, I'm not on the same system I'd do such work from.

> but it may be easier to just work with
> ssh -v -v -v azure-server-git-server git-upload-pack /url
>
>
> I'm afraid I have no pony for you, Nico. My guess is that the CIS
> hardening removed something (e.g. ssh-rsa) which is the only one
> supported by that Azure git service, but you surely had a similar guess
> already.

That's what I suspect. I was hoping someone else with broad SSH
experience might already have the answer in hand. Microsoft was not
helpful in answering which specific protocols their SSH daemon
supports, which I did try to ask.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 21.05.22 01:41, Nico Kadel-Garcia wrote:
> I'm dealing with Azure DevOps git services with which recent, security
> hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> based SSH links to the Azure Devops. Other RHEL 8 based systems work
> well.
>
> Conversely, Azure Bastion cannot use Azure key vault stored private
> SSH keys to access the same RHEL 8 based servers, though they can
> reach other hosts with the same private key and the same public key.
> I've also used the private SSH key, locally, to connect to the various
> servers.

In my experience, if you can run a plain "ssh -v" as the client, the
output *does* include details about the problem *somewhere*. To wit:

> $ ssh -v root@SomeOldBox
[...]
> Unable to negotiate with SomeOldBox port 22: no matching key exchange > method found. Their offer: diffie-hellman-group-exchange-sha1,>
diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

> $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1" root@SomeOldBox
[...]
> Unable to negotiate with SomeOldBox port 22: no matching
> host key type found. Their offer: ssh-rsa,ssh-dss

> $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1"
> -o "HostKeyAlgorithms +ssh-rsa" root@SomeOldBox
[...]
> Unable to negotiate with SomeOldBox port 22: no matching MAC found.
> Their offer: hmac-md5,hmac-sha1,hmac-ripemd160,
> hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
etc. etc.. If I had to *guess*, I'd say that the dissent is about the
key vault somehow offering (only) ssh-rsa while the hardened RHEL now
insists on rsa-sha2-* for a key algorithm ...

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

On Mon, May 23, 2022 at 5:07 AM Jochen Bern <Jochen.Bern@binect.de> wrote:
>
> On 21.05.22 01:41, Nico Kadel-Garcia wrote:
> > I'm dealing with Azure DevOps git services with which recent, security
> > hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> > based SSH links to the Azure Devops. Other RHEL 8 based systems work
> > well.
> >
> > Conversely, Azure Bastion cannot use Azure key vault stored private
> > SSH keys to access the same RHEL 8 based servers, though they can
> > reach other hosts with the same private key and the same public key.
> > I've also used the private SSH key, locally, to connect to the various
> > servers.

I figured out the Bastion problem. Generating keys and getting them
into the Azure key vault correctly can be a bit of an adventure. It
turns out that the "az keyvault" commands can get very peculiar about
handling quotes, and of course the "insert punctuation in new
passwords" requirements of Azure AD and it's linked ADDS can be....
adventuresome to script safely and consistently for integration with
configuration management tools.

The Azure DevOps git service issues are still a problem. I've not so
far found the distinction between the InterNIC published RHEL 8 images
and the Red Hat published RHEL 8 images that trigger this issue. I do
like I do like your suggestion of poking the 'ssh -v -o
"KexAlgorithms +diffie-hellman-group14-sha1" -o "HostKeyAlgorithms
+ssh-rsa" ' settings, and will do so ASAP. That thought.... makes
sense.

Nico Kadel-Garcia

> In my experience, if you can run a plain "ssh -v" as the client, the
> output *does* include details about the problem *somewhere*. To wit:
>
> > $ ssh -v root@SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching key exchange > method found. Their offer: diffie-hellman-group-exchange-sha1,>
> diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>
> > $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1" root@SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching
> > host key type found. Their offer: ssh-rsa,ssh-dss
>
> > $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1"
> > -o "HostKeyAlgorithms +ssh-rsa" root@SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching MAC found.
> > Their offer: hmac-md5,hmac-sha1,hmac-ripemd160,
> > hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> etc. etc.. If I had to *guess*, I'd say that the dissent is about the
> key vault somehow offering (only) ssh-rsa while the hardened RHEL now
> insists on rsa-sha2-* for a key algorithm ...
>
> Regards,
> --
> Jochen Bern
> Systemingenieur
>
> Binect GmbH
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Mon, May 23, 2022 at 5:07 AM Jochen Bern <Jochen.Bern@binect.de> wrote:
>
> On 21.05.22 01:41, Nico Kadel-Garcia wrote:
> > I'm dealing with Azure DevOps git services with which recent, security
> > hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> > based SSH links to the Azure Devops. Other RHEL 8 based systems work
> > well.
> >
> > Conversely, Azure Bastion cannot use Azure key vault stored private
> > SSH keys to access the same RHEL 8 based servers, though they can
> > reach other hosts with the same private key and the same public key.
> > I've also used the private SSH key, locally, to connect to the various
> > servers.
>
> In my experience, if you can run a plain "ssh -v" as the client, the
> output *does* include details about the problem *somewhere*. To wit:
>
> > $ ssh -v root@SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching key exchange > method found. Their offer: diffie-hellman-group-exchange-sha1,>
> diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>
> > $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1" root@SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching
> > host key type found. Their offer: ssh-rsa,ssh-dss
>
> > $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1"
> > -o "HostKeyAlgorithms +ssh-rsa" root@SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching MAC found.
> > Their offer: hmac-md5,hmac-sha1,hmac-ripemd160,
> > hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
> etc. etc.. If I had to *guess*, I'd say that the dissent is about the
> key vault somehow offering (only) ssh-rsa while the hardened RHEL now
> insists on rsa-sha2-* for a key algorithm ...

The key vault stores whatever key I put in it. The issue is that on
the CIS published OpenSSH client setup, they no longer support plain
"ssh-rsa" protocol anymore. The trick is to enable it as needed:

Host azure-git-server.domain
PuubKeyAcceptedKeyTypes +ssh-rsa

The safest place to put it for all clients on the server is:

/etc/ssh/ssh_config.d/10-azure-git-server

I'm unsure of the change is built into the most recent versions of
OpenSSH on RHEL 8, or in another package like crypto-policies that was
hand-modified by the authors of the particular VM. But it's working
well now.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev