Mailing List Archive

older versions of OpenSSH periodically kill the connection when multiple channels are opened
Sometimes when I connect to older OpenSSH servers the server closes the
connection. Doing "sudo /usr/sbin/sshd -ddd -p2222 &" gives me the
following error:

> debug1: Unable to open session: The name org.freedesktop.ConsoleKit was
not provided by any .service files

What the client is doing is... it's opening up an interactive session on
one channel, requesting a pty and starting a shell on that channel and then
running "ping 127.0.0.1" on the shell.

The client then opens up another interactive session, on another channel
and tries to request a pty on the other channel but no response is received
because the server has closed the connection.

Here's the full log of the back and forth unencrypted SSH traffic:

-> SSH_MSG_CHANNEL_OPEN (since last: 1.9427, network: 0s)
00000000 00:00:00:07:73:65:73:73:69:6f:6e:00:00:00:02:7f ....session.....
00000010 ff:ff:ff:00:00:40:00 .....@.

<- SSH_MSG_CHANNEL_OPEN_CONFIRMATION (since last: 0.0355, network: 0.0001s)
00000000 00:00:00:02:00:00:00:00:00:00:00:00:00:00:80:00 ................

-> SSH_MSG_CHANNEL_REQUEST (since last: 0.0003, network: 0s)
00000000 00:00:00:00:00:00:00:07:70:74:79:2d:72:65:71:01 ........pty-req.
00000010 00:00:00:05:76:74:31:30:30:00:00:00:50:00:00:00 ....vt100...P...
00000020 18:00:00:00:00:00:00:00:00:00:00:00:01:00 ..............

<- SSH_MSG_CHANNEL_SUCCESS (since last: 0.0006, network: 0s)
00000000 00:00:00:02 ....

-> SSH_MSG_CHANNEL_REQUEST (since last: 0.0002, network: 0s)
00000000 00:00:00:00:00:00:00:05:73:68:65:6c:6c:01 ........shell.

<- SSH_MSG_CHANNEL_WINDOW_ADJUST (since last: 0.0006, network: 0s)
00000000 00:00:00:02:00:20:00:00 ..... ..

<- SSH_MSG_CHANNEL_SUCCESS (since last: 0.0001, network: 0s)
00000000 00:00:00:02 ....

-> SSH_MSG_CHANNEL_DATA (since last: 0.0002, network: 0s)
00000000 00:00:00:00:00:00:00:0f:70:69:6e:67:20:31:32:37 ........ping 127
00000010 2e:30:2e:30:2e:31:0a .0.0.1.

-> SSH_MSG_CHANNEL_OPEN (since last: 0.0002, network: 0s)
00000000 00:00:00:07:73:65:73:73:69:6f:6e:00:00:00:01:7f ....session.....
00000010 ff:ff:ff:00:00:40:00 .....@.

<- SSH_MSG_CHANNEL_DATA (since last: 0.0005, network: 0s)
00000000 00:00:00:02:00:00:00:36:4c:61:73:74:20:6c:6f:67 .......6Last log
00000010 69:6e:3a:20:54:75:65:20:4f:63:74:20:31:32:20:30 in: Tue Oct 12 0
00000020 33:3a:34:33:3a:31:31:20:32:30:32:31:20:66:72:6f 3:43:11 2021 fro
00000030 6d:20:6c:6f:63:61:6c:68:6f:73:74:0d:0d:0a m localhost...

<- SSH_MSG_CHANNEL_OPEN_CONFIRMATION (since last: 0.0003, network: 0s)
00000000 00:00:00:01:00:00:00:01:00:00:00:00:00:00:80:00 ................

-> SSH_MSG_CHANNEL_REQUEST (since last: 0.0002, network: 0s)
00000000 00:00:00:01:00:00:00:07:70:74:79:2d:72:65:71:01 ........pty-req.
00000010 00:00:00:05:76:74:31:30:30:00:00:00:50:00:00:00 ....vt100...P...
00000020 18:00:00:00:00:00:00:00:00:00:00:00:01:00 ..............

Here's the full error log that the OpenSSH server is providing:

debug1: server_input_channel_req: channel 1 request pty-req reply 1
debug1: session_by_channel: session 1 channel 1
debug1: session_input_channel_req: session 1 req pty-req
debug1: Allocating pty.
debug3: mm_request_send entering: type 28
debug3: mm_pty_allocate: waiting for MONITOR_ANS_PTY
debug3: mm_request_receive_expect entering: type 29
debug3: mm_request_receive entering
debug1: Unable to open session: The name org.freedesktop.ConsoleKit was not
provided by any .service files
debug3: mm_request_send entering: type 157
debug3: mm_request_receive entering
debug3: monitor_read: checking request 28
debug3: mm_answer_pty entering
debug2: session_new: allocate (allocated 1 max 10)
debug3: session_unused: session id 1 unused
debug1: session_new: session 1
debug3: mm_request_send entering: type 29
debug3: mm_answer_pty: tty /dev/pts/3 ptyfd 9
mm_request_receive_expect: read: rtype 157 != type 29
debug1: do_cleanup
debug3: PAM: sshpam_thread_cleanup entering
debug3: mm_request_receive entering
mm_request_receive: read: Connection reset by peer
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials
debug3: PAM: sshpam_thread_cleanup entering
debug1: session_pty_cleanup: session 0 release /dev/pts/2
debug1: unregistering ConsoleKit session (null)
debug1: session_pty_cleanup: session 1 release /dev/pts/3
debug1: audit_event: unhandled event 12

I haven't seen this issue on SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 or
later versions but, none-the-less, this error is causing some unit tests to
fail, intermittently.

Should I make it so that the client won't open up multiple channels on
versions of OpenSSH older than 7.2?

If this issue is indeed a resolved issue is there a changelog entry that
mentions the fix that I can cite in my code?
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: older versions of OpenSSH periodically kill the connection when multiple channels are opened [ In reply to ]
On Wed, 13 Oct 2021 at 14:07, Terra Frost <terrafrost@gmail.com> wrote:
> Sometimes when I connect to older OpenSSH servers the server closes the
> connection. Doing "sudo /usr/sbin/sshd -ddd -p2222 &" gives me the
> following error:
>
> > debug1: Unable to open session: The name org.freedesktop.ConsoleKit was
> not provided by any .service files

Exactly which version are you seeing this with? AFAICT no version of
OpenSSH shipped by the OpenSSH team contains that debug message.

> Here's the full error log that the OpenSSH server is providing:

This is not the full log, if it was the sshd version identifier would
be present.

[...]
> debug1: Allocating pty.
> debug3: mm_request_send entering: type 28

From monitor.h this is MONITOR_REQ_PTY.

> debug3: mm_pty_allocate: waiting for MONITOR_ANS_PTY
> debug3: mm_request_receive_expect entering: type 29
> debug3: mm_request_receive entering

29 is indeed MONITOR_ANS_PTY

> debug1: Unable to open session: The name org.freedesktop.ConsoleKit was not
provided by any .service files

as mentioned above this message is not present in stock OpenSSH.

> debug3: mm_request_send entering: type 157

monitor.h does not define a type 157. Debian's gssapi patch defines
up to 153 and its selinux patch defines 154 so this is something else
again.

> debug3: mm_request_receive entering
> debug3: monitor_read: checking request 28
> debug3: mm_answer_pty entering
> debug2: session_new: allocate (allocated 1 max 10)
> debug3: session_unused: session id 1 unused
> debug1: session_new: session 1
> debug3: mm_request_send entering: type 29
> debug3: mm_answer_pty: tty /dev/pts/3 ptyfd 9
> mm_request_receive_expect: read: rtype 157 != type 29

This looks to me like the monitor and the unprivileged process are
getting out of sync although it's not clear why.

> debug1: unregistering ConsoleKit session (null)

this is also not from the stock OpenSSH code.

> I haven't seen this issue on SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 or
> later versions but, none-the-less, this error is causing some unit tests to
> fail, intermittently.

That's a version that doesn't have the problem. Which versions do
have the problem?

> Should I make it so that the client won't open up multiple channels on
> versions of OpenSSH older than 7.2?

That's up to you, but it looks like this problem is specific to a
vendor modified version.

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: older versions of OpenSSH periodically kill the connection when multiple channels are opened [ In reply to ]
On Wed, 13 Oct 2021 at 15:03, Darren Tucker <dtucker@dtucker.net> wrote:
[...]
> monitor.h does not define a type 157. Debian's gssapi patch defines
> up to 153 and its selinux patch defines 154 so this is something else
> again.

Looks like this is Debian's ConsoleKit patch:

https://sources.debian.org/patches/openssh/1:6.7p1-5+deb8u4/consolekit.patch/

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: older versions of OpenSSH periodically kill the connection when multiple channels are opened [ In reply to ]
On Wed, 13 Oct 2021 at 15:09, Darren Tucker <dtucker@dtucker.net> wrote:
>
> On Wed, 13 Oct 2021 at 15:03, Darren Tucker <dtucker@dtucker.net> wrote:
> [...]
> > monitor.h does not define a type 157. Debian's gssapi patch defines
> > up to 153 and its selinux patch defines 154 so this is something else
> > again.
>
> Looks like this is Debian's ConsoleKit patch:
>
> https://sources.debian.org/patches/openssh/1:6.7p1-5+deb8u4/consolekit.patch/

and assuming that's it, the problem is that the patch calls a monitor
function from do_setup_env(0 which is called from do_child(), which is
in the process forked off to run the user's shell (but before it's
exec'ed the shell). You're not supposed to do that, and it causes a
race condition when the parent (which *is* supposed to do that) is
making monitor calls at the same time. Depending on which order the
monitor requests are sent in and which order the child processes get
CPU and read the responses you might see this or the inverse failure
("rtype 29 != type 157") or see it work.

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: older versions of OpenSSH periodically kill the connection when multiple channels are opened [ In reply to ]
https://pastebin.com/q5KFKmWm is the full log in case you're curious. But
the
https://sources.debian.org/patches/openssh/1:6.7p1-5+deb8u4/consolekit.patch/
link looks like a good find!

On Tue, Oct 12, 2021 at 11:03 PM Darren Tucker <dtucker@dtucker.net> wrote:

> On Wed, 13 Oct 2021 at 14:07, Terra Frost <terrafrost@gmail.com> wrote:
> > Sometimes when I connect to older OpenSSH servers the server closes the
> > connection. Doing "sudo /usr/sbin/sshd -ddd -p2222 &" gives me the
> > following error:
> >
> > > debug1: Unable to open session: The name org.freedesktop.ConsoleKit was
> > not provided by any .service files
>
> Exactly which version are you seeing this with? AFAICT no version of
> OpenSSH shipped by the OpenSSH team contains that debug message.
>
> > Here's the full error log that the OpenSSH server is providing:
>
> This is not the full log, if it was the sshd version identifier would
> be present.
>
> [...]
> > debug1: Allocating pty.
> > debug3: mm_request_send entering: type 28
>
> From monitor.h this is MONITOR_REQ_PTY.
>
> > debug3: mm_pty_allocate: waiting for MONITOR_ANS_PTY
> > debug3: mm_request_receive_expect entering: type 29
> > debug3: mm_request_receive entering
>
> 29 is indeed MONITOR_ANS_PTY
>
> > debug1: Unable to open session: The name org.freedesktop.ConsoleKit was
> not
> provided by any .service files
>
> as mentioned above this message is not present in stock OpenSSH.
>
> > debug3: mm_request_send entering: type 157
>
> monitor.h does not define a type 157. Debian's gssapi patch defines
> up to 153 and its selinux patch defines 154 so this is something else
> again.
>
> > debug3: mm_request_receive entering
> > debug3: monitor_read: checking request 28
> > debug3: mm_answer_pty entering
> > debug2: session_new: allocate (allocated 1 max 10)
> > debug3: session_unused: session id 1 unused
> > debug1: session_new: session 1
> > debug3: mm_request_send entering: type 29
> > debug3: mm_answer_pty: tty /dev/pts/3 ptyfd 9
> > mm_request_receive_expect: read: rtype 157 != type 29
>
> This looks to me like the monitor and the unprivileged process are
> getting out of sync although it's not clear why.
>
> > debug1: unregistering ConsoleKit session (null)
>
> this is also not from the stock OpenSSH code.
>
> > I haven't seen this issue on SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 or
> > later versions but, none-the-less, this error is causing some unit tests
> to
> > fail, intermittently.
>
> That's a version that doesn't have the problem. Which versions do
> have the problem?
>
> > Should I make it so that the client won't open up multiple channels on
> > versions of OpenSSH older than 7.2?
>
> That's up to you, but it looks like this problem is specific to a
> vendor modified version.
>
> --
> Darren Tucker (dtucker at dtucker.net)
> GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
> Good judgement comes with experience. Unfortunately, the experience
> usually comes from bad judgement.
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev