Mailing List Archive

Question about sshbuf
I’m trying to build a version of OpenSSH 9.0p1 with the GSSAPI patch at https://sources.debian.org/patches/openssh/1:9.0p1-1/gssapi.patch/. The patch applied just fine and I had no issues with the build or the tests (all of which passed).

That said, I think I’ve discovered a bug in the patch. I know that’s not the responsibility of the OpenSSH maintainers, but I was hoping to get some help from this list on a higher-level question about the right way to work with sshbufs, as I haven’t had a lot of experience actually making changes to OpenSSH.

The issue occurs when the GSS client kex code receives a KEXGSS_HOSTKEY message. There’s code to handle this message, but it runs into an issue where it reports "ssh_packet_read: read: internal error: buffer is read-only”. When I looked into this more closely, the issue is not actually that the sshbuf it is using is read-only. It’s that the sshbuf has a refcount greater than 1 at the time the code tries to reuse it. Here’s the code in question:

struct sshbuf *server_host_key_blob = NULL;

...snip...
/* If we've sent them data, they should reply */
do {
type = ssh_packet_read(ssh);
if (type == SSH2_MSG_KEXGSS_HOSTKEY) {
debug("Received KEXGSS_HOSTKEY");
if (server_host_key_blob)
fatal("Server host key received more than once");
if ((r = sshpkt_getb_froms(ssh, &server_host_key_blob)) != 0)
fatal("Failed to read server host key: %s", ssh_err(r));
}
} while (type == SSH2_MSG_KEXGSS_HOSTKEY);
...snip...
out:
sshbuf_free(server_host_key_blob);

I believe the problem here is that the call to sshpkt_getb_froms() is returning an sshbuf in server_host_key_blob which is a reference to the string being consumed from the packet being read, setting that original packet as its parent. As a result, the “ssh” buffer now has a refcount of 2, and when it returns to the top of the do {...} while and tries to read another packet into “ssh”, it gets the error about the sshbuf being “read-only” (for good reason).

There’s already code in place to free server_host_key_blob, so I think the solution here is probably to make a copy of this blob into a new sshbuf rather than getting a reference to the existing memory. However, I’m not sure what the best function is to use in place of sshpkt_getb_froms() which will guarantee a new allocation is performed. Anyone have a suggestion on this? The sshbuf_get_stringb() seems close, but it doesn’t take into account that “ssh” needs to be accessed as a packet. I see various sshpkt_get_string functions, but not specifically an sshpkt_get_stringb(). It wouldn’t be hard to add, but if it’s not already present there’s probably a good reason for that. Is there a better way to do this?

As an aside, I’m guessing this refcounting capability was added after the patch was originally written, and the patch was never updated to consider this issue, meaning there might be other places in the patch where this occurs. Is there anything I can read that might provide more information about which functions began returning references rather than always making copies of buffers?

Thanks in advance for any help you can provide!
--
Ron Frederick
ronf@timeheart.net



_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On Sat, 21 May 2022, Ron Frederick wrote:

> I’m trying to build a version of OpenSSH 9.0p1 with the GSSAPI patch
> at https://sources.debian.org/patches/openssh/1:9.0p1-1/gssapi.patch/.
> The patch applied just fine and I had no issues with the build or the
> tests (all of which passed).
>
> That said, I think I’ve discovered a bug in the patch. I know that’s
> not the responsibility of the OpenSSH maintainers, but I was hoping
> to get some help from this list on a higher-level question about the
> right way to work with sshbufs, as I haven’t had a lot of experience
> actually making changes to OpenSSH.
>
> The issue occurs when the GSS client kex code receives a
> KEXGSS_HOSTKEY message. There’s code to handle this message, but it
> runs into an issue where it reports "ssh_packet_read: read: internal
> error: buffer is read-only”. When I looked into this more closely, the
> issue is not actually that the sshbuf it is using is read-only. It’s
> that the sshbuf has a refcount greater than 1 at the time the code
> tries to reuse it. Here’s the code in question:
>
> struct sshbuf *server_host_key_blob = NULL;
>
> ...snip...
> /* If we've sent them data, they should reply */
> do {
> type = ssh_packet_read(ssh);
> if (type == SSH2_MSG_KEXGSS_HOSTKEY) {
> debug("Received KEXGSS_HOSTKEY");
> if (server_host_key_blob)
> fatal("Server host key received more than once");
> if ((r = sshpkt_getb_froms(ssh, &server_host_key_blob)) != 0)
> fatal("Failed to read server host key: %s", ssh_err(r));
> }
> } while (type == SSH2_MSG_KEXGSS_HOSTKEY);
> ...snip...
> out:
> sshbuf_free(server_host_key_blob);
>
> I believe the problem here is that the call to sshpkt_getb_froms() is
> returning an sshbuf in server_host_key_blob which is a reference to
> the string being consumed from the packet being read, setting that
> original packet as its parent. As a result, the “ssh” buffer now has
> a refcount of 2, and when it returns to the top of the do {...} while
> and tries to read another packet into “ssh”, it gets the error about
> the sshbuf being “read-only” (for good reason).

IMO the best fix for this would be to put a sshbuf_free() at the
start of the do {} loop (it will handle a NULL argument fine).
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On 5/22/22 02:17, Ron Frederick wrote:
> I’m trying to build a version of OpenSSH 9.0p1 with the GSSAPI patch at https://sources.debian.org/patches/openssh/1:9.0p1-1/gssapi.patch/. The patch applied just fine and I had no issues with the build or the tests (all of which passed).
>
> That said, I think I’ve discovered a bug in the patch. I know that’s not the responsibility of the OpenSSH maintainers, but I was hoping to get some help from this list on a higher-level question about the right way to work with sshbufs, as I haven’t had a lot of experience actually making changes to OpenSSH.
>
> The issue occurs when the GSS client kex code receives a KEXGSS_HOSTKEY message. There’s code to handle this message, but it runs into an issue where it reports "ssh_packet_read: read: internal error: buffer is read-only”. When I looked into this more closely, the issue is not actually that the sshbuf it is using is read-only. It’s that the sshbuf has a refcount greater than 1 at the time the code tries to reuse it. Here’s the code in question:
>
> struct sshbuf *server_host_key_blob = NULL;
>
> ...snip...
> /* If we've sent them data, they should reply */
> do {
> type = ssh_packet_read(ssh);
> if (type == SSH2_MSG_KEXGSS_HOSTKEY) {
> debug("Received KEXGSS_HOSTKEY");
> if (server_host_key_blob)
> fatal("Server host key received more than once");
> if ((r = sshpkt_getb_froms(ssh, &server_host_key_blob)) != 0)
> fatal("Failed to read server host key: %s", ssh_err(r));
> }
> } while (type == SSH2_MSG_KEXGSS_HOSTKEY);
> ...snip...
> out:
> sshbuf_free(server_host_key_blob);
>
> I believe the problem here is that the call to sshpkt_getb_froms() is returning an sshbuf in server_host_key_blob which is a reference to the string being consumed from the packet being read, setting that original packet as its parent. As a result, the “ssh” buffer now has a refcount of 2, and when it returns to the top of the do {...} while and tries to read another packet into “ssh”, it gets the error about the sshbuf being “read-only” (for good reason).
>
> There’s already code in place to free server_host_key_blob, so I think the solution here is probably to make a copy of this blob into a new sshbuf rather than getting a reference to the existing memory. However, I’m not sure what the best function is to use in place of sshpkt_getb_froms() which will guarantee a new allocation is performed. Anyone have a suggestion on this? The sshbuf_get_stringb() seems close, but it doesn’t take into account that “ssh” needs to be accessed as a packet. I see various sshpkt_get_string functions, but not specifically an sshpkt_get_stringb(). It wouldn’t be hard to add, but if it’s not already present there’s probably a good reason for that. Is there a better way to do this?
>
> As an aside, I’m guessing this refcounting capability was added after the patch was originally written, and the patch was never updated to consider this issue, meaning there might be other places in the patch where this occurs. Is there anything I can read that might provide more information about which functions began returning references rather than always making copies of buffers?
>
> Thanks in advance for any help you can provide!

We track the gsskex patches in the following github repository:

https://github.com/openssh-gsskex/openssh-gsskex/

I did not read into details about that, but believe I already saw this
issue and we were fixing it:

https://github.com/openssh-gsskex/openssh-gsskex/pull/19

Unfortunately, the repository is not completely up to date, but both
Colin and Dmitry should be able to help you around here.

Regards,
--
Jakub Jelen
Crypto Team, Security Engineering
Red Hat, Inc.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On May 22, 2022, at 7:03 PM, Damien Miller <djm@mindrot.org> wrote:
> On Sat, 21 May 2022, Ron Frederick wrote:
>> I’m trying to build a version of OpenSSH 9.0p1 with the GSSAPI patch
>> at https://sources.debian.org/patches/openssh/1:9.0p1-1/gssapi.patch/.
>> The patch applied just fine and I had no issues with the build or the
>> tests (all of which passed).
>>
>> That said, I think I’ve discovered a bug in the patch. I know that’s
>> not the responsibility of the OpenSSH maintainers, but I was hoping
>> to get some help from this list on a higher-level question about the
>> right way to work with sshbufs, as I haven’t had a lot of experience
>> actually making changes to OpenSSH.
>>
>> The issue occurs when the GSS client kex code receives a
>> KEXGSS_HOSTKEY message. There’s code to handle this message, but it
>> runs into an issue where it reports "ssh_packet_read: read: internal
>> error: buffer is read-only”. When I looked into this more closely, the
>> issue is not actually that the sshbuf it is using is read-only. It’s
>> that the sshbuf has a refcount greater than 1 at the time the code
>> tries to reuse it. Here’s the code in question:
>>
>> struct sshbuf *server_host_key_blob = NULL;
>>
>> ...snip...
>> /* If we've sent them data, they should reply */
>> do {
>> type = ssh_packet_read(ssh);
>> if (type == SSH2_MSG_KEXGSS_HOSTKEY) {
>> debug("Received KEXGSS_HOSTKEY");
>> if (server_host_key_blob)
>> fatal("Server host key received more than once");
>> if ((r = sshpkt_getb_froms(ssh, &server_host_key_blob)) != 0)
>> fatal("Failed to read server host key: %s", ssh_err(r));
>> }
>> } while (type == SSH2_MSG_KEXGSS_HOSTKEY);
>> ...snip...
>> out:
>> sshbuf_free(server_host_key_blob);
>>
>> I believe the problem here is that the call to sshpkt_getb_froms() is
>> returning an sshbuf in server_host_key_blob which is a reference to
>> the string being consumed from the packet being read, setting that
>> original packet as its parent. As a result, the “ssh” buffer now has
>> a refcount of 2, and when it returns to the top of the do {...} while
>> and tries to read another packet into “ssh”, it gets the error about
>> the sshbuf being “read-only” (for good reason).
>
> IMO the best fix for this would be to put a sshbuf_free() at the
> start of the do {} loop (it will handle a NULL argument fine).


Thanks Damien for the quick response.

Given that “ssh” is actually a “struct ssh *” and not a “struct sshbuf *”, what would be the right argument to pass into sshbuf_free()? It looks like it would need to be something like ssh->state->incoming_packet, but this calling code probably shouldn’t be poking into the internals like that. I don’t see a wrapper in packet.c which would do a free of that internal buffer.
--
Ron Frederick
ronf@timeheart.net



_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On Mon, 23 May 2022, Ron Frederick wrote:

> >> ...snip...
> >> out:
> >> sshbuf_free(server_host_key_blob);
> >>
> >> I believe the problem here is that the call to sshpkt_getb_froms() is
> >> returning an sshbuf in server_host_key_blob which is a reference to
> >> the string being consumed from the packet being read, setting that
> >> original packet as its parent. As a result, the “ssh” buffer now has
> >> a refcount of 2, and when it returns to the top of the do {...} while
> >> and tries to read another packet into “ssh”, it gets the error about
> >> the sshbuf being “read-only” (for good reason).
> >
> > IMO the best fix for this would be to put a sshbuf_free() at the
> > start of the do {} loop (it will handle a NULL argument fine).
>
>
> Thanks Damien for the quick response.
>
> Given that “ssh” is actually a “struct ssh *” and not a “struct sshbuf *”, what would be the right argument to pass into sshbuf_free()? It looks like it would need to be something like ssh->state->incoming_packet, but this calling code probably shouldn’t be poking into the internals like that. I don’t see a wrapper in packet.c which would do a free of that internal buffer.

I'd do this:

> >> /* If we've sent them data, they should reply */
> >> do {

sshbuf_free(server_host_key_blob);
server_host_key_blob = NULL;

> >> type = ssh_packet_read(ssh);
> >> if (type == SSH2_MSG_KEXGSS_HOSTKEY) {
> >> debug("Received KEXGSS_HOSTKEY");
> >> if (server_host_key_blob)
> >> fatal("Server host key received more than once");
> >> if ((r = sshpkt_getb_froms(ssh, &server_host_key_blob)) != 0)
> >> fatal("Failed to read server host key: %s", ssh_err(r));
> >> }
> >> } while (type == SSH2_MSG_KEXGSS_HOSTKEY);
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On May 23, 2022, at 3:55 PM, Damien Miller <djm@mindrot.org> wrote:
> On Mon, 23 May 2022, Ron Frederick wrote:
>>>> ...snip...
>>>> out:
>>>> sshbuf_free(server_host_key_blob);
>>>>
>>>> I believe the problem here is that the call to sshpkt_getb_froms() is
>>>> returning an sshbuf in server_host_key_blob which is a reference to
>>>> the string being consumed from the packet being read, setting that
>>>> original packet as its parent. As a result, the “ssh” buffer now has
>>>> a refcount of 2, and when it returns to the top of the do {...} while
>>>> and tries to read another packet into “ssh”, it gets the error about
>>>> the sshbuf being “read-only” (for good reason).
>>>
>>> IMO the best fix for this would be to put a sshbuf_free() at the
>>> start of the do {} loop (it will handle a NULL argument fine).
>>
>>
>> Thanks Damien for the quick response.
>>
>> Given that “ssh” is actually a “struct ssh *” and not a “struct sshbuf *”, what would be the right argument to pass into sshbuf_free()? It looks like it would need to be something like ssh->state->incoming_packet, but this calling code probably shouldn’t be poking into the internals like that. I don’t see a wrapper in packet.c which would do a free of that internal buffer.
>
> I'd do this:
>
>>>> /* If we've sent them data, they should reply */
>>>> do {
>
> sshbuf_free(server_host_key_blob);
> server_host_key_blob = NULL;
>
>>>> type = ssh_packet_read(ssh);
>>>> if (type == SSH2_MSG_KEXGSS_HOSTKEY) {
>>>> debug("Received KEXGSS_HOSTKEY");
>>>> if (server_host_key_blob)
>>>> fatal("Server host key received more than once");
>>>> if ((r = sshpkt_getb_froms(ssh, &server_host_key_blob)) != 0)
>>>> fatal("Failed to read server host key: %s", ssh_err(r));
>>>> }
>>>> } while (type == SSH2_MSG_KEXGSS_HOSTKEY);


I don’t think that will work, as server_host_key_blob is used outside of the do..while loop. It needs to survive for the lifetime of the call to kexgss_client(). See line 282 for where it is used.
--
Ron Frederick
ronf@timeheart.net



_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On May 23, 2022, at 12:46 AM, Jakub Jelen <jjelen@redhat.com> wrote:
> I believe the problem here is that the call to sshpkt_getb_froms() is returning an sshbuf in server_host_key_blob which is a reference to the string being consumed from the packet being read, setting that original packet as its parent. As a result, the “ssh” buffer now has a refcount of 2, and when it returns to the top of the do {...} while and tries to read another packet into “ssh”, it gets the error about the sshbuf being “read-only” (for good reason).
>
> We track the gsskex patches in the following github repository:
>
> https://github.com/openssh-gsskex/openssh-gsskex/
>
> I did not read into details about that, but believe I already saw this issue and we were fixing it:
>
> https://github.com/openssh-gsskex/openssh-gsskex/pull/19
>
> Unfortunately, the repository is not completely up to date, but both Colin and Dmitry should be able to help you around here.


Thanks, Jakub! I did find that Github page, but went looking for the 9.0p1 Debian patch because it seemed more up to date than the Git repo. As you mentioned, it seems like that Git repo is based on something like 8.3p1, though there is an outstanding rebase to 8.8p1 which would get it closer.

The pull request you mention here is also still waiting to be integrated. I appreciate the pointer to that, though — it suggests a possible approach to handling the memory management using sshpkt_get_string() and sshbuf_from(). I’ll give it a try!

I appreciate the help...
--
Ron Frederick
ronf@timeheart.net



_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On May 23, 2022, at 4:30 PM, Ron Frederick <ronf@timeheart.net> wrote:
> On May 23, 2022, at 12:46 AM, Jakub Jelen <jjelen@redhat.com> wrote:
>> I believe the problem here is that the call to sshpkt_getb_froms() is returning an sshbuf in server_host_key_blob which is a reference to the string being consumed from the packet being read, setting that original packet as its parent. As a result, the “ssh” buffer now has a refcount of 2, and when it returns to the top of the do {...} while and tries to read another packet into “ssh”, it gets the error about the sshbuf being “read-only” (for good reason).
>>
>> We track the gsskex patches in the following github repository:
>>
>> https://github.com/openssh-gsskex/openssh-gsskex/
>>
>> I did not read into details about that, but believe I already saw this issue and we were fixing it:
>>
>> https://github.com/openssh-gsskex/openssh-gsskex/pull/19
>>
>> Unfortunately, the repository is not completely up to date, but both Colin and Dmitry should be able to help you around here.
>
> Thanks, Jakub! I did find that Github page, but went looking for the 9.0p1 Debian patch because it seemed more up to date than the Git repo. As you mentioned, it seems like that Git repo is based on something like 8.3p1, though there is an outstanding rebase to 8.8p1 which would get it closer.
>
> The pull request you mention here is also still waiting to be integrated. I appreciate the pointer to that, though — it suggests a possible approach to handling the memory management using sshpkt_get_string() and sshbuf_from(). I’ll give it a try!


Looks like the patch in https://github.com/openssh-gsskex/openssh-gsskex/pull/19 <https://github.com/openssh-gsskex/openssh-gsskex/pull/19> applied cleanly on top of the Debian 9.0p1 patch I used, and it fixes the issue I was seeing with the KEXGSS_HOSTKEY message. I’m no longer seeing a fatal error there about a read-only buffer, and both the GSS key exchange and GSS keyex authentication appear to be working. I also tested GSS MIC authentication, and that appears to be still be working as well, both with & without GSS key exchange.

Thanks again everyone for the help!

The only remaining issue I’m seeing is around getting this patched OpenSSH to offer gssapi-keyex as an auth method when running it as a server. Even when it correctly negotiates GSS key exchange, it doesn’t offer gssapi-keyex as a valid auth method to clients, so they end up falling back to gssapi-with-mic instead. This only happens when running it as a server — if another server offers gssapi-keyex, this patched OpenSSH client has no problem completing gssapi-keyex auth. I’ll take a closer look at that next.
--
Ron Frederick
ronf@timeheart.net



_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Question about sshbuf [ In reply to ]
On May 23, 2022, at 5:01 PM, Ron Frederick <ronf@timeheart.net> wrote:
> The only remaining issue I’m seeing is around getting this patched OpenSSH to offer gssapi-keyex as an auth method when running it as a server. Even when it correctly negotiates GSS key exchange, it doesn’t offer gssapi-keyex as a valid auth method to clients, so they end up falling back to gssapi-with-mic instead. This only happens when running it as a server — if another server offers gssapi-keyex, this patched OpenSSH client has no problem completing gssapi-keyex auth. I’ll take a closer look at that next.


I found this issue tonight as well. It turns out that the Authmethod type recently added a “synonym” member, but the definition for the GSS keyex auth method was not updated to include this field when the Debian patch was put together, causing the members after it to be initiated with the wrong values. Thankfully, the fix is very simple:

--- auth2-gss.c.orig 2022-05-23 19:49:36.000000000 -0700
+++ auth2-gss.c 2022-05-23 20:46:17.000000000 -0700
@@ -373,6 +373,7 @@

Authmethod method_gsskeyex = {
"gssapi-keyex",
+ NULL,
userauth_gsskeyex,
&options.gss_authentication
};

It looks like this change was made in https://github.com/openssh/openssh-portable/commit/dbb339f <https://github.com/openssh/openssh-portable/commit/dbb339f> and also includes adding a second argument to the userauth functions, so really the diff should probably be:

--- auth2-gss.c.orig 2022-05-23 19:49:36.000000000 -0700
+++ auth2-gss.c 2022-05-23 20:57:45.000000000 -0700
@@ -59,7 +59,7 @@
* The 'gssapi_keyex' userauth mechanism.
*/
static int
-userauth_gsskeyex(struct ssh *ssh)
+userauth_gsskeyex(struct ssh *ssh, const char *method)
{
Authctxt *authctxt = ssh->authctxt;
int r, authenticated = 0;
@@ -373,6 +373,7 @@

Authmethod method_gsskeyex = {
"gssapi-keyex",
+ NULL,
userauth_gsskeyex,
&options.gss_authentication
};

The commit which added this was from December of 2021, so it probably didn’t show up until 8.9p1. As a result, this change wouldn’t apply to the version at https://github.com/openssh-gsskex/openssh-gsskex/ <https://github.com/openssh-gsskex/openssh-gsskex/> yet.

With this fix, gssapi-keyex authentication is now properly enabled and functional for both the client and server.
--
Ron Frederick
ronf@timeheart.net



_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev