Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. OpenSSH>
  3. Dev
"Bad packet length 1231976033"
philipp_subx at redfish-solutions
Apr 8, 2023, 6:20 PM
Post #1 of 13 (777 views)
Permalink
I have a cron script that polls various machines for their configs and archives any changes.

When it connects to an OpenWrt router running "OpenSSH_9.1p1, OpenSSL 1.1.1s 1 Nov 2022", it sometimes gets this error:

Bad packet length 1231976033.
ssh_dispatch_run_fatal: Connection to 192.168.8.1 port 22: message authentication code incorrect

What's odd is that the length is *always* 1231976033 (which is 0x496E7661 or "Inva" in ASCII).

The client is Ubuntu 22.04.2 LTS, and it's version is "OpenSSH_9.1p1, OpenSSL 3.0.2 15 Mar 2022".

I googled this and there are various hypothesis like it's a mismatch in protocol negotiation versions (both client and server will only do Version 2); that my .bashrc file can't emit any output on the server (my server uses Ash); that there can't be an /etc/motd file; that I need to set 'LC_ALL="en_US.UTF-8"' when invoking ssh; and all sorts of other theories which seemed to be clutching at straws.

So, what's the real fix?

Thanks,

-Philip

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
b.candler at pobox
Apr 9, 2023, 12:21 AM
Post #2 of 13 (777 views)
Permalink
On 09/04/2023 02:20, Philip Prindeville wrote:
> What's odd is that the length is*always* 1231976033 (which is 0x496E7661 or "Inva" in ASCII).

Could you get a tcpdump when this happens? Then maybe more of the error
can be captured.

I grepped for Inva in the source code. There are lots of error messages
which start with this which are sent with error() or fatal() or
fprint(stderr, ...), but two which stand out as being a bit different:

# kex.c

 invalid:
                send_error(ssh, "Invalid SSH identification string.");

# packet.c

        if (*typep < SSH2_MSG_MIN || *typep >= SSH2_MSG_LOCAL_MIN) {
                if ((r = sshpkt_disconnect(ssh,
                    "Invalid ssh2 packet type: %d", *typep)) != 0 ||

If it happens to be either of these, then the tcpdump showing the
exchange prior to this point would be enlightening.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
peter at stuge
Apr 9, 2023, 1:57 PM
Post #3 of 13 (777 views)
Permalink
Brian Candler wrote:
> > What's odd is that the length is *always* 1231976033 (which is
> > 0x496E7661 or "Inva" in ASCII).
>
> Could you get a tcpdump when this happens?

Or debug output from at least the client (run ssh with -vvv) or
preferably the server (run sshd with -ddd).


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
dtucker at dtucker
Apr 10, 2023, 6:24 AM
Post #4 of 13 (776 views)
Permalink
On Mon, 10 Apr 2023 at 07:07, Peter Stuge <peter@stuge.se> wrote:
>
> Brian Candler wrote:
> > > What's odd is that the length is *always* 1231976033 (which is
> > > 0x496E7661 or "Inva" in ASCII).

One thing that can cause this is if the libc writes to stderr (ie fd
2) on some classes of error. This is something libc should probably
not do, since in things that are not simple command line tools (say, a
ssh daemon) may be using fd 2 for something else entirely.

> > Could you get a tcpdump when this happens?
>
> Or debug output from at least the client (run ssh with -vvv) or
> preferably the server (run sshd with -ddd).

That's probably not going to show it, but strace'ing either the client
or the server will probably capture the error message in full.

Since you're using 9.1, the message could be an "Invalid free", since
there was a double-free bug in that release :-(

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
philipp_subx at redfish-solutions
Apr 23, 2023, 1:08 PM
Post #5 of 13 (762 views)
Permalink
Sorry about taking so long to get back to you. The problem is sporadic and I've had other fires to put out first...

Here's a PCAP of authentication failures:

https://www.redfish-solutions.com/misc/kvm1.pcap



> On Apr 9, 2023, at 1:21 AM, Brian Candler <b.candler@pobox.com> wrote:
>
> On 09/04/2023 02:20, Philip Prindeville wrote:
>> What's odd is that the length is*always* 1231976033 (which is 0x496E7661 or "Inva" in ASCII).
>
> Could you get a tcpdump when this happens? Then maybe more of the error can be captured.
>
> I grepped for Inva in the source code. There are lots of error messages which start with this which are sent with error() or fatal() or fprint(stderr, ...), but two which stand out as being a bit different:
>
> # kex.c
>
> invalid:
> send_error(ssh, "Invalid SSH identification string.");
>
> # packet.c
>
> if (*typep < SSH2_MSG_MIN || *typep >= SSH2_MSG_LOCAL_MIN) {
> if ((r = sshpkt_disconnect(ssh,
> "Invalid ssh2 packet type: %d", *typep)) != 0 ||
>
> If it happens to be either of these, then the tcpdump showing the exchange prior to this point would be enlightening.
>

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
b.candler at pobox
Apr 24, 2023, 12:09 AM
Post #6 of 13 (762 views)
Permalink
On 23/04/2023 21:08, Philip Prindeville wrote:
> Here's a PCAP of authentication failures:
>
> https://www.redfish-solutions.com/misc/kvm1.pcap

Although I see some instances 49 6e, and some of 76 61, I don't see any
49 6e 76 61. If it's in there, it must be encrypted.

I think it will be necessary to do as others have said: run sshd with
debugging cranked up.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
philipp_subx at redfish-solutions
Apr 24, 2023, 10:17 AM
Post #7 of 13 (760 views)
Permalink
> On Apr 10, 2023, at 7:24 AM, Darren Tucker <dtucker@dtucker.net> wrote:
>
> On Mon, 10 Apr 2023 at 07:07, Peter Stuge <peter@stuge.se> wrote:
>>
>> Brian Candler wrote:
>>>> What's odd is that the length is *always* 1231976033 (which is
>>>> 0x496E7661 or "Inva" in ASCII).
>
> One thing that can cause this is if the libc writes to stderr (ie fd
> 2) on some classes of error. This is something libc should probably
> not do, since in things that are not simple command line tools (say, a
> ssh daemon) may be using fd 2 for something else entirely.


Agreed. libc or libcrypto...


>
>>> Could you get a tcpdump when this happens?
>>
>> Or debug output from at least the client (run ssh with -vvv) or
>> preferably the server (run sshd with -ddd).
>
> That's probably not going to show it, but strace'ing either the client
> or the server will probably capture the error message in full.
>
> Since you're using 9.1, the message could be an "Invalid free", since
> there was a double-free bug in that release :-(
>


Ah, good to know.

One other thing I noticed: I have ssh issues to/from my Ubuntu VM or slow https downloads until I reboot it, then it works properly for a while until it doesn't.

Wondering if it's also a possible device driver related issue? None of the other guests (OpenWrt, CentOS, Fedora, etc) seem to have network related issues. Just Ubuntu. And it happens when Ubuntu is either the client or the server (at least for ssh/scp).

-Philip

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
philipp_subx at redfish-solutions
Apr 24, 2023, 10:36 AM
Post #8 of 13 (760 views)
Permalink
> On Apr 10, 2023, at 7:24 AM, Darren Tucker <dtucker@dtucker.net> wrote:
>
> On Mon, 10 Apr 2023 at 07:07, Peter Stuge <peter@stuge.se> wrote:
>>
>> Brian Candler wrote:
>>>> What's odd is that the length is *always* 1231976033 (which is
>>>> 0x496E7661 or "Inva" in ASCII).
>
> One thing that can cause this is if the libc writes to stderr (ie fd
> 2) on some classes of error. This is something libc should probably
> not do, since in things that are not simple command line tools (say, a
> ssh daemon) may be using fd 2 for something else entirely.
>
>>> Could you get a tcpdump when this happens?
>>
>> Or debug output from at least the client (run ssh with -vvv) or
>> preferably the server (run sshd with -ddd).
>
> That's probably not going to show it, but strace'ing either the client
> or the server will probably capture the error message in full.
>
> Since you're using 9.1, the message could be an "Invalid free", since
> there was a double-free bug in that release :-(
>


Forgot to ask: does this bug manifest at any particular time, or just connection initiation? Because I can see it happen on a connection that's been up for days... either idle or experiencing heavy traffic... etc.

-Philip


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
peter at stuge
Apr 24, 2023, 1:11 PM
Post #9 of 13 (760 views)
Permalink
Philip Prindeville wrote:
> One other thing I noticed: I have ssh issues to/from my Ubuntu VM
> or slow https downloads until I reboot it, then it works properly
> for a while until it doesn't.
>
> Wondering if it's also a possible device driver related issue?
> None of the other guests (OpenWrt, CentOS, Fedora, etc) seem to
> have network related issues. Just Ubuntu. And it happens when
> Ubuntu is either the client or the server (at least for ssh/scp).

I guess that it's related to distribution patching rather than device
drivers. Try using unmodified openssh-portable code on that ubuntu
system and see if that's any different.

Also, what's the connection between "https downloads" and SSH?


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
dtucker at dtucker
Apr 24, 2023, 5:22 PM
Post #10 of 13 (760 views)
Permalink
On Tue, 25 Apr 2023 at 03:36, Philip Prindeville
<philipp_subx@redfish-solutions.com> wrote:
> > On Apr 10, 2023, at 7:24 AM, Darren Tucker <dtucker@dtucker.net> wrote:
[...]
> > Since you're using 9.1, the message could be an "Invalid free", since
> > there was a double-free bug in that release :-(
>
> Forgot to ask: does this bug manifest at any particular time, or just connection initiation?
> Because I can see it happen on a connection that's been up for days... either idle or experiencing heavy traffic... etc.

Only at connection initiation, and only when the client is an old
version of PuTTY that doesn't support the modern form of the
diffie-hellman group exchange protocol.

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
philipp_subx at redfish-solutions
Apr 24, 2023, 5:29 PM
Post #11 of 13 (760 views)
Permalink
> On Apr 24, 2023, at 6:22 PM, Darren Tucker <dtucker@dtucker.net> wrote:
>
> On Tue, 25 Apr 2023 at 03:36, Philip Prindeville
> <philipp_subx@redfish-solutions.com> wrote:
>>> On Apr 10, 2023, at 7:24 AM, Darren Tucker <dtucker@dtucker.net> wrote:
> [...]
>>> Since you're using 9.1, the message could be an "Invalid free", since
>>> there was a double-free bug in that release :-(
>>
>> Forgot to ask: does this bug manifest at any particular time, or just connection initiation?
>> Because I can see it happen on a connection that's been up for days... either idle or experiencing heavy traffic... etc.
>
> Only at connection initiation, and only when the client is an old
> version of PuTTY that doesn't support the modern form of the
> diffie-hellman group exchange protocol.
>


Okay. I'm using Openssh 9.1p1 between Ubuntu and OpenWrt, or Openssh 9.0p1 on MacOS 13.3.1.


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
philipp_subx at redfish-solutions
Jul 20, 2023, 11:08 AM
Post #12 of 13 (601 views)
Permalink
> On Apr 24, 2023, at 6:29 PM, Philip Prindeville <philipp_subx@redfish-solutions.com> wrote:
>
>
>
>> On Apr 24, 2023, at 6:22 PM, Darren Tucker <dtucker@dtucker.net> wrote:
>>
>> On Tue, 25 Apr 2023 at 03:36, Philip Prindeville
>> <philipp_subx@redfish-solutions.com> wrote:
>>>> On Apr 10, 2023, at 7:24 AM, Darren Tucker <dtucker@dtucker.net> wrote:
>> [...]
>>>> Since you're using 9.1, the message could be an "Invalid free", since
>>>> there was a double-free bug in that release :-(
>>>
>>> Forgot to ask: does this bug manifest at any particular time, or just connection initiation?
>>> Because I can see it happen on a connection that's been up for days... either idle or experiencing heavy traffic... etc.
>>
>> Only at connection initiation, and only when the client is an old
>> version of PuTTY that doesn't support the modern form of the
>> diffie-hellman group exchange protocol.
>>
>
>
> Okay. I'm using Openssh 9.1p1 between Ubuntu and OpenWrt, or Openssh 9.0p1 on MacOS 13.3.1.



Huh. This is interesting.

When I moved my KVM *host* OS for my guest VM's (including Ubuntu and OpenWrt) from CentOS 7.9 to RHEL 9.1, the problem seems to have gone away, though not RCA available.

My guess is that the host network drivers were corrupting certain packets, though... I was using SR-IOV so there should have been very little manipulation of the ring-buffers by the host OS, other than stripping the VLAN headers off of them (since the VLAN headers were apparent to the host's hostdev-net interface, but the guest was unaware of them).

-Philip

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: "Bad packet length 1231976033" [ In reply to ]
philipp_subx at redfish-solutions
Jul 20, 2023, 11:12 AM
Post #13 of 13 (601 views)
Permalink
> On Apr 24, 2023, at 2:11 PM, Peter Stuge <peter@stuge.se> wrote:
>
> Philip Prindeville wrote:
>> One other thing I noticed: I have ssh issues to/from my Ubuntu VM
>> or slow https downloads until I reboot it, then it works properly
>> for a while until it doesn't.
>>
>> Wondering if it's also a possible device driver related issue?
>> None of the other guests (OpenWrt, CentOS, Fedora, etc) seem to
>> have network related issues. Just Ubuntu. And it happens when
>> Ubuntu is either the client or the server (at least for ssh/scp).
>
> I guess that it's related to distribution patching rather than device
> drivers. Try using unmodified openssh-portable code on that ubuntu
> system and see if that's any different.
>
> Also, what's the connection between "https downloads" and SSH?
>
>
> //Peter


Sorry, maybe that wasn't clear.

HTTP/S and HTTP downloads on the clients (such as "apt-get update") would get slower and slower until I rebooted the Ubuntu guest VM's.

Per my other posting a couple of minutes ago, moving KVM from a CentOS 7.9 host (and the ancient version of KVM, Qemu, and libvirt that comes with that distro) to RHEL 9.1 resolved a lot of problems.

I'm running 5.15 kernel which is not stellar, but it's good enough.

-Philip

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal