Mailing List Archive

Upstream forwarding test failure
Dear colleagues,

I came across an upstream test suite failure on Fedora 36.
The test in question is forwarding, the output is

==========
adding modulifile='/home/dbelyavs/work/upstream/openssh-portable/moduli' to
sshd_config
using cached key type ssh-ed25519
using cached key type sk-ssh-ed25519@openssh.com
using cached key type ecdsa-sha2-nistp256
using cached key type ecdsa-sha2-nistp384
using cached key type ecdsa-sha2-nistp521
using cached key type sk-ecdsa-sha2-nistp256@openssh.com
using cached key type ssh-dss
using cached key type ssh-rsa
wait for sshd
start forwarding, fork to background
transfer over forwarded channels and check result
exit on -L forward failure
connection not termintated, but should (0)
exit on -R forward failure
connection not termintated, but should (0)
simple clear forwarding
clear local forward
clear remote forward
stdio forwarding
config file: start forwarding, fork to background
config file: transfer over forwarded channels and check result
transfer over chained unix domain socket forwards and check result
wait for sshd to exit
failed local and remote forwarding
make[1]: *** [Makefile:224: t-exec] Error 1
make[1]: Leaving directory
'/home/dbelyavs/work/upstream/openssh-portable/regress'
make: *** [Makefile:724: t-exec] Error 2
==========

The test (especially built from upstream) shouldn't be affected by Fedora
cryptography limitations. Unfortunately I also can't get any clues from
regress/failed_ssh.log

Any help will be appreciated. Many thanks in advance!

--
Dmitry Belyavskiy
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Upstream forwarding test failure [ In reply to ]
On Wed, 25 Jan 2023 at 01:38, Dmitry Belyavskiy <dbelyavs@redhat.com> wrote:
[...]
> The test (especially built from upstream) shouldn't be affected by Fedora
> cryptography limitations. Unfortunately I also can't get any clues from
> regress/failed_ssh.log

Unfortunately we don't do a good job with debug logging for the tests
where there's more than one ssh/sshd pair, and the forwarding tests
like this one are one example of that.

I have a part-done patch that logs the output from all ssh and sshd
instances to separate datestamped files. I'll see if I can tidy that
up for you to try, but in the meantime check to see if there's
anything else listening in range of TEST_SSH_PORT (4242 by default)
and up.

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Upstream forwarding test failure [ In reply to ]
On Wed, 25 Jan 2023 at 19:29, Darren Tucker <dtucker@dtucker.net> wrote:
[...]
> I have a part-done patch that logs the output from all ssh and sshd
> instances to separate datestamped files. I'll see if I can tidy that
> up for you to try

You can grab it from here:
https://github.com/daztucker/openssh-portable/commit/b54b39349e1a64cbbb9b56b0f8b91a35589fb528

It's not fully baked, but it should at least give you some more
insight into the case you are looking at:

$ make t-exec LTESTS=forwarding

and check the logs in regress/log (the oldest and newest are likely to
be the most interesting).

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Upstream forwarding test failure [ In reply to ]
Dear Darren,

On Wed, Jan 25, 2023 at 10:13 AM Darren Tucker <dtucker@dtucker.net> wrote:

> On Wed, 25 Jan 2023 at 19:29, Darren Tucker <dtucker@dtucker.net> wrote:
> [...]
> > I have a part-done patch that logs the output from all ssh and sshd
> > instances to separate datestamped files. I'll see if I can tidy that
> > up for you to try
>
> You can grab it from here:
>
> https://github.com/daztucker/openssh-portable/commit/b54b39349e1a64cbbb9b56b0f8b91a35589fb528
>
> It's not fully baked, but it should at least give you some more
> insight into the case you are looking at:
>
> $ make t-exec LTESTS=forwarding
>
> and check the logs in regress/log (the oldest and newest are likely to
> be the most interesting).
>
>
Do I correctly understand that the idea of the failing test here [1] is
that we try to bind
to the same host/port using IP and hostname, and it should fail as we don't
set
SO_REUSEADDR/SO_REUSEPORT socket options?

If yes, than I suspect some problem on the level of my system setup because
the log looks like
===============
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to 127.0.0.1 ([127.0.0.1]:4242) using "publickey".
debug1: Remote connections from LOCALHOST:3301 forwarded to local address
127.0.0.1:4242
debug3: send packet: type 80
debug1: Remote connections from LOCALHOST:3302 forwarded to local address
127.0.0.1:4242
debug3: send packet: type 80
debug1: Remote connections from LOCALHOST:3303 forwarded to local address
127.0.0.1:4242
debug3: send packet: type 80
debug1: Remote connections from LOCALHOST:3301 forwarded to local address
localhost:4242
debug3: send packet: type 80
debug1: Remote connections from LOCALHOST:3304 forwarded to local address
127.0.0.1:4242
debug3: send packet: type 80
debug1: ssh_init_forwarding: expecting replies for 5 forwards
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: client_repledge: enter
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com
want_reply 0
debug3: receive packet: type 4
debug1: Remote:
/home/dbelyavs/work/upstream/openssh-portable/regress/authorized_keys_dbelyavs:1:
key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote:
/home/dbelyavs/work/upstream/openssh-portable/regress/authorized_keys_dbelyavs:1:
key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 81
debug1: remote forward success for: listen 3301, connect 127.0.0.1:4242
debug2: forwarding_success: 4 expected forwarding replies remaining
debug3: receive packet: type 81
debug1: remote forward success for: listen 3302, connect 127.0.0.1:4242
debug2: forwarding_success: 3 expected forwarding replies remaining
debug3: receive packet: type 81
debug1: remote forward success for: listen 3303, connect 127.0.0.1:4242
debug2: forwarding_success: 2 expected forwarding replies remaining
debug3: receive packet: type 81
debug1: remote forward success for: listen 3301, connect localhost:4242
debug2: forwarding_success: 1 expected forwarding replies remaining
debug3: receive packet: type 81
debug1: remote forward success for: listen 3304, connect 127.0.0.1:4242
debug1: forwarding_success: all expected forwarding replies received
debug3: receive packet: type 91
================
so the second forwarding succeeds.

I have no ideas how to debug it further, so any help is much appropriate!

[1]
https://github.com/openssh/openssh-portable/blob/master/regress/forwarding.sh#L52

--
Dmitry Belyavskiy
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev