Mailing List Archive

[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

JM <jtm.moon.forum.user+mindrot@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Summary|server thread aborts during |server thread aborts during
|client login after |client login after
|receiving SSH2_MSG_KEXINIT |receiving SSH2_MSG_KEXINIT
| |due to
| |ssh_sandbox_violation

--
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

Damien Miller <djm@mindrot.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |djm@mindrot.org

--- Comment #4 from Damien Miller <djm@mindrot.org> ---
This is the details of the sandbox violation:

> ssh_sandbox_violation: unexpected system call (arch:0x40000028,syscall:20 @ 0xf7ba380c

syscall 20 is getpid:

> [djm@djm linux]$ grep 'NR.* 20$' arch/arm64/include/asm/unistd32.h
> #define __NR_getpid 20

but getpid is allowed by the sshd sandbox policy:

> [djm@djm openssh]$ grep -A1 getpid sandbox-seccomp-filter.c
> #ifdef __NR_getpid
> SC_ALLOW(__NR_getpid),
> #endif

However, this only works in __NR_getpid is defined in a system header
than that header is correctly picked up during sshd's compilation. If
your system headers are messed up then sshd won't pick up the correct
syscall number and sandbox violations will occur.

You could try attaching ./configure output and config.h (please don't
paste them inline), which might help, but I suspect that the root cause
is going that your system headers are messed up in some way.

--
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #5 from Darren Tucker <dtucker@dtucker.net> ---
(In reply to Damien Miller from comment #4)
> This is the details of the sandbox violation:
>
> > ssh_sandbox_violation: unexpected system call (arch:0x40000028,syscall:20 @ 0xf7ba380c
>
> syscall 20 is getpid:
>
> > [djm@djm linux]$ grep 'NR.* 20$' arch/arm64/include/asm/unistd32.h
> > #define __NR_getpid 20

That's not what it is on my rpi4. I think that's for 32bit ARM.

$ uname -a
Linux hostname 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST
2023 aarch64 GNU/Linux
$ grep -r -E '__NR.*20$' /usr/include
/usr/include/asm-generic/unistd.h:#define __NR_epoll_create1 20

$ grep -E NR.*getpid /usr/include/asm-generic/unistd.h
#define __NR_getpid 172

which a test program confirms:

$ cat test.c
#include <stdio.h>
#include <syscall.h>
int main(void)
{
printf("__NR_epoll_create1 %d\n", __NR_epoll_create1);
printf("__NR_getpid %d\n", __NR_getpid);
}
$ cc test.c && ./a.out
__NR_epoll_create1 20
__NR_getpid 172

Testing on a 32bit arm, that is indeed 20:
$ uname -a
Linux hostname 5.16.10-bone14 #1bullseye PREEMPT Tue Feb 22 00:07:39
UTC 2022 armv7l GNU/Linux

$ cc test.c && ./a.out
__NR_epoll_create1 357
__NR_getpid 20

So perhaps the problem here is that either it's picking up 32bit vs
64bit headers, or that the binary is some kind of 32bit compatibility
mode but the sandbox is expecting the 64bit syscalls.

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #6 from Darren Tucker <dtucker@dtucker.net> ---
JM: what compiler are you using?

Another thing that might be interesting is to compare config.h and the
output of configure from 9.1p1 with those of 9.2p1 and see if anything
unexpectedly changed.

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #7 from Darren Tucker <dtucker@dtucker.net> ---
Created attachment 3764
--> https://bugzilla.mindrot.org/attachment.cgi?id=3764&action=edit
configure output from 9.2p1 working on rpi4

--
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #8 from Darren Tucker <dtucker@dtucker.net> ---
Created attachment 3765
--> https://bugzilla.mindrot.org/attachment.cgi?id=3765&action=edit
config.h from 9.2p1 working on rpi4

here's the configure output and config.h from my working system for
comparison.

--
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #9 from Damien Miller <djm@mindrot.org> ---
hmm, it looks like I might have been wrong with the syscall number:

> [djm@djm linux]$ grep ' 20$' include/uapi/asm-generic/unistd.h
> #define __NR_epoll_create1 20

It's likely that something is trying to use the epoll(3) API. OpenSSH
itself doesn't use epoll, so it's likely to be something in libc,
libcrypto or another library.

--
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #10 from Darren Tucker <dtucker@dtucker.net> ---
(In reply to Damien Miller from comment #9)
> It's likely that something is trying to use the epoll(3) API.
> OpenSSH itself doesn't use epoll, so it's likely to be something in
> libc, libcrypto or another library.

That's possible. I checked /etc/os-release on my device, and it's
stock Debian not Raspbian. My guess would be something in libcrypto,
in which case configuring --without-openssl and retesting would be a
good indicator).

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #11 from Darren Tucker <dtucker@dtucker.net> ---
I'd like to reproduce this locally. Could you please attach
/etc/os-release and the output of "dpkg -l" from the affected device?

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #22 from Damien Miller <djm@mindrot.org> ---
Created attachment 3774
--> https://bugzilla.mindrot.org/attachment.cgi?id=3774&action=edit
syscall dumper

Could you try building and running this program. E.g.

$ cc -o syscall syscall.c
$ ./syscall
$ strace -n ./syscall

This will tell you what the compiler thinks __NR_getpid is vs what the
actual syscall is.

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #23 from Damien Miller <djm@mindrot.org> ---
i.e. run it on a platform that works and the one that doesn't

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #24 from JM <jtm.moon.forum.user+mindrot@gmail.com> ---
> Could you try building and running this program. E.g.
>
> $ cc -o syscall syscall.c
> $ ./syscall
> $ strace -n ./syscall


Raspberry Pi 4 (RPi4), aarch64, Raspbian-Debian 11, (openssh 9.5p1
client thread aborts):

$ (set -eux; wget -q
"https://bugzilla.mindrot.org/attachment.cgi?id=3774" -O syscall.c ; cc
-o syscall syscall.c ; ./syscall ; strace -n ./syscall )
+ wget -q 'https://bugzilla.mindrot.org/attachment.cgi?id=3774' -O
syscall.c
+ cc -o syscall syscall.c
+ ./syscall
__NR_getpid = 20
+ strace -n ./syscall
[ 11] execve("./syscall", ["./syscall"], 0xffb2b5b4 /* 31 vars */)
= 0
[ 45] brk(NULL) = 0x1ae5000
[ 122] uname({sysname="Linux", nodename="pifuboo", ...}) = 0
[ 33] access("/etc/ld.so.preload", R_OK) = 0
[ 322] openat(AT_FDCWD, "/etc/ld.so.preload",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 197] fstat64(3, {st_mode=S_IFREG|0644, st_size=54, ...}) = 0
[ 192] mmap2(NULL, 54, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) =
0xf7d7f000
[ 6] close(3) = 0
[ 85] readlink("/proc/self/exe", "/tmp/syscall", 4096) = 12
[ 322] openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 3] read(3,
"\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\254\3\0\0004\0\0\0"...,
512) = 512
[ 197] fstat64(3, {st_mode=S_IFREG|0644, st_size=17708, ...}) = 0
[ 192] mmap2(NULL, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf7d7d000
[ 192] mmap2(NULL, 81964, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xf7d3a000
[ 125] mprotect(0xf7d3e000, 61440, PROT_NONE) = 0
[ 192] mmap2(0xf7d4d000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0xf7d4d000
[ 6] close(3) = 0
[ 91] munmap(0xf7d7f000, 54) = 0
[ 322] openat(AT_FDCWD, "/etc/ld.so.cache",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 197] fstat64(3, {st_mode=S_IFREG|0644, st_size=63463, ...}) = 0
[ 192] mmap2(NULL, 63463, PROT_READ, MAP_PRIVATE, 3, 0) =
0xf7d2a000
[ 6] close(3) = 0
[ 322] openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libc.so.6",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 3] read(3,
"\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\0y\1\0004\0\0\0"...,
512) = 512
[ 197] fstat64(3, {st_mode=S_IFREG|0755, st_size=1315688, ...}) = 0
[ 192] mmap2(NULL, 1385020, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xf7bd7000
[ 125] mprotect(0xf7d15000, 61440, PROT_NONE) = 0
[ 192] mmap2(0xf7d24000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13d000) = 0xf7d24000
[ 192] mmap2(0xf7d27000, 8764, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xf7d27000
[ 6] close(3) = 0
[983045] set_tls(0xf7d7df80) = 0
[ 125] mprotect(0xf7d24000, 8192, PROT_READ) = 0
[ 125] mprotect(0xf7d4d000, 4096, PROT_READ) = 0
[ 125] mprotect(0x20000, 4096, PROT_READ) = 0
[ 125] mprotect(0xf7d81000, 4096, PROT_READ) = 0
[ 91] munmap(0xf7d2a000, 63463) = 0
[ 197] fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0),
...}) = 0
[ 45] brk(NULL) = 0x1ae5000
[ 45] brk(0x1b06000) = 0x1b06000
[ 4] write(1, "__NR_getpid = 20\n", 17__NR_getpid = 20
) = 17
[ 20] getpid() = 21435
[ 248] exit_group(0) = ?
[ 248] +++ exited with 0 +++

Raspberry Pi 3 (RPi3), armv7l, Raspbian Debian 11, (openssh 9.5p1 runs
okay):

$ (set -eux; wget -q
"https://bugzilla.mindrot.org/attachment.cgi?id=3774" -O syscall.c ; cc
-o syscall syscall.c ; ./syscall ; strace -n ./syscall )
+ wget -q 'https://bugzilla.mindrot.org/attachment.cgi?id=3774' -O
syscall.c
+ cc -o syscall syscall.c
+ ./syscall
__NR_getpid = 20
+ strace -n ./syscall
[ 11] execve("./syscall", ["./syscall"], 0x7eab2584 /* 30 vars */)
= 0
[ 45] brk(NULL) = 0x1435000
[ 192] mmap2(NULL, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76fa5000
[ 33] access("/etc/ld.so.preload", R_OK) = 0
[ 322] openat(AT_FDCWD, "/etc/ld.so.preload",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 197] fstat64(3, {st_mode=S_IFREG|0644, st_size=54, ...}) = 0
[ 192] mmap2(NULL, 54, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) =
0x76fa4000
[ 6] close(3) = 0
[ 85] readlink("/proc/self/exe", "/tmp/syscall", 4096) = 12
[ 322] openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 3] read(3,
"\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\254\3\0\0004\0\0\0"...,
512) = 512
[ 197] fstat64(3, {st_mode=S_IFREG|0644, st_size=17708, ...}) = 0
[ 192] mmap2(NULL, 81964, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76f60000
[ 125] mprotect(0x76f64000, 61440, PROT_NONE) = 0
[ 192] mmap2(0x76f73000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x76f73000
[ 6] close(3) = 0
[ 91] munmap(0x76fa4000, 54) = 0
[ 322] openat(AT_FDCWD, "/etc/ld.so.cache",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 197] fstat64(3, {st_mode=S_IFREG|0644, st_size=55352, ...}) = 0
[ 192] mmap2(NULL, 55352, PROT_READ, MAP_PRIVATE, 3, 0) =
0x76f97000
[ 6] close(3) = 0
[ 322] openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libc.so.6",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
[ 3] read(3,
"\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\0y\1\0004\0\0\0"...,
512) = 512
[ 197] fstat64(3, {st_mode=S_IFREG|0755, st_size=1315688, ...}) = 0
[ 192] mmap2(NULL, 1385020, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76e0d000
[ 125] mprotect(0x76f4b000, 61440, PROT_NONE) = 0
[ 192] mmap2(0x76f5a000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13d000) = 0x76f5a000
[ 192] mmap2(0x76f5d000, 8764, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76f5d000
[ 6] close(3) = 0
[983045] set_tls(0x76fa6280) = 0
[ 125] mprotect(0x76f5a000, 8192, PROT_READ) = 0
[ 125] mprotect(0x76f73000, 4096, PROT_READ) = 0
[ 125] mprotect(0x20000, 4096, PROT_READ) = 0
[ 125] mprotect(0x76fa7000, 4096, PROT_READ) = 0
[ 91] munmap(0x76f97000, 55352) = 0
[ 197] fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0),
...}) = 0
[ 45] brk(NULL) = 0x1435000
[ 45] brk(0x1456000) = 0x1456000
[ 4] write(1, "__NR_getpid = 20\n", 17__NR_getpid = 20
) = 17
[ 20] getpid() = 19383
[ 248] exit_group(0) = ?
[ 248] +++ exited with 0 +++

NanoPi NEO3 Rockchip RK3288, aarch64, Debian, (openssh 9.5p1 runs
okay):

$ (set -eux; wget -q
"https://bugzilla.mindrot.org/attachment.cgi?id=3774" -O syscall.c ; cc
-o syscall syscall.c ; ./syscall ; strace -n ./syscall )
+ wget -q 'https://bugzilla.mindrot.org/attachment.cgi?id=3774' -O
syscall.c
+ cc -o syscall syscall.c
+ ./syscall
__NR_getpid = 172
+ strace -n ./syscall
[ 221] execve("./syscall", ["./syscall"], 0xffffed56ea98 /* 27 vars
*/) = 0
[ 214] brk(NULL) = 0xaaaae9498000
[ 48] faccessat(AT_FDCWD, "/etc/ld.so.preload", R_OK) = -1 ENOENT
(No such file or directory)
[ 56] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[ 80] fstat(3, {st_mode=S_IFREG|0644, st_size=24779, ...}) = 0
[ 222] mmap(NULL, 24779, PROT_READ, MAP_PRIVATE, 3, 0) =
0xffffa86b7000
[ 57] close(3) = 0
[ 56] openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libc.so.6",
O_RDONLY|O_CLOEXEC) = 3
[ 63] read(3,
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0`\17\2\0\0\0\0\0"...,
832) = 832
[ 80] fstat(3, {st_mode=S_IFREG|0755, st_size=1451024, ...}) = 0
[ 222] mmap(NULL, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa86b5000
[ 222] mmap(NULL, 1523656, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffffa851c000
[ 226] mprotect(0xffffa8678000, 61440, PROT_NONE) = 0
[ 222] mmap(0xffffa8687000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15b000) = 0xffffa8687000
[ 222] mmap(0xffffa868d000, 12232, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffffa868d000
[ 57] close(3) = 0
[ 226] mprotect(0xffffa8687000, 16384, PROT_READ) = 0
[ 226] mprotect(0xaaaaacbf0000, 4096, PROT_READ) = 0
[ 226] mprotect(0xffffa86c1000, 4096, PROT_READ) = 0
[ 215] munmap(0xffffa86b7000, 24779) = 0
[ 80] fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0),
...}) = 0
[ 214] brk(NULL) = 0xaaaae9498000
[ 214] brk(0xaaaae94b9000) = 0xaaaae94b9000
[ 64] write(1, "__NR_getpid = 172\n", 18__NR_getpid = 172
) = 18
[ 172] getpid() = 7407
[ 94] exit_group(0) = ?
[ 94] +++ exited with 0 +++

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #25 from Damien Miller <djm@mindrot.org> ---
ok, so now I have no idea what is going wrong. Maybe there is something
in OpenSSH's compile flags that is messing this up.

Could you add try adding a similar printf+getpid+exit sequence to (say)
the start of ssh-keygen.c, rerunning make and repeating the test using
the resultant binary? You probably only need to do this on the rpi4

--
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #26 from JM <jtm.moon.forum.user+mindrot@gmail.com> ---
tl;dr a seccomp sandbox violation `20` occurs from a `read` (still).
This is just a more detailed retelling of what was previously
discussed.
Scroll to end for thoughts...

### problem specifics

Failed `read` in parent process

read(fd, s + pos, n - pos);

which is

read(5, '\014\221b', 4);

returns `0`.

Failed `read` will cause the following audit event (journalctl -f -x)

Dec 17 15:59:35 host1 kernel: audit: type=1326
audit(1702857575.824:3180): auid=4294967295 uid=107 gid=65534
ses=4294967295 pid=1920 comm="sshd"
exe="/root/Projects/openssh-9.2p1-WIP/sshd" sig=31 arch=40000028
syscall=20 compat=1 ip=0xf7afd80c code=0x0

And the same when compiled with
`CFLAGS="-DDSANDBOX_SECCOMP_FILTER_DEBUG"`

Dec 17 22:37:50 pifuboo audit[10678]: SECCOMP auid=4294967295
uid=107 gid=65534 ses=4294967295 pid=10678 comm="sshd"
exe="/root/Projects/openssh-9.2p1-WIP/sshd" sig=31 arch=40000028
syscall=20 compat=1 ip=0xf77de80c code=0x0
Dec 17 22:37:50 pifuboo audit[10678]: ANOM_ABEND auid=4294967295
uid=107 gid=65534 ses=4294967295 pid=10678 comm="sshd"
exe="/root/Projects/openssh-9.2p1-WIP/sshd" sig=31 res=1

The failed linux system call `20` is `epoll_create1` according to
`ausyscall`

$ ausyscall 20
epoll_create1

So the `read` at some point calls syscall `20`. See section "Summary
Thoughts" about this.

Here is the failed `read` call
https://github.com/openssh/openssh-portable/blob/V_9_2_P1/atomicio.c#L66
It is always the `read` call with values `fd=5`, `n=4`.
`read` returns `0`.
`errno` is not changed after `read` returns.

The stack just before the failed `read` call is:

#1 0x004701f8 in atomicio6
(f=f@entry=0xf7c65478 <read>, fd=fd@entry=5, _s=0xfffeead8,
_s@entry=0xfffeead0, n=n@entry=4, cb=cb@entry=0x0,
cb_arg=cb_arg@entry=0x0)
at atomicio.c:67
#2 0x00470284 in atomicio
(f=f@entry=0xf7c65478 <read>, fd=fd@entry=5,
_s=_s@entry=0xfffeead0, n=n@entry=4)
at atomicio.c:101
#3 0x00434520 in mm_request_receive
(sock=5, m=m@entry=0x4f2b88)
at monitor_wrap.c:149
#4 0x00431178 in monitor_read
(ssh=ssh@entry=0x4f3388, pmonitor=pmonitor@entry=0x4f2498,
ent=0x4e0114 <mon_dispatch_proto20>, pent=pent@entry=0xfffeeb78)
at monitor.c:501
#5 0x00433b5c in monitor_child_preauth
(ssh=ssh@entry=0x4f3388, pmonitor=0x4f2498)
at monitor.c:301
#6 0x0040a388 in privsep_preauth
(ssh=0x4f3388)
at sshd.c:502
#7 main
(ac=<optimized out>, av=0x4e31a0)
at sshd.c:2240

(line numbers in `atomicio.c` slightly off due to insertion of
`raise(SIGINT);`)

The debug log from the start of client connection is:

debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 3296
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
process 32277 is executing new program:
/root/Projects/openssh-9.2p1/sshd
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/arm-linux-gnueabihf/libthread_db.so.1".
debug3: recv_rexec_state: entering fd = 5
debug3: ssh_msg_recv entering
debug3: recv_rexec_state: done
debug2: parse_server_config_depth: config rexec len 3296
debug3: rexec:14 setting Port 55222
debug3: rexec:22 setting HostKey
/root/Projects/openssh-9.2p1/ssh_host_ecdsa_key
debug3: rexec:23 setting HostKey
/root/Projects/openssh-9.2p1/ssh_host_ed25519_key
debug3: rexec:24 setting HostKey
/root/Projects/openssh-9.2p1/ssh_host_rsa_key
debug3: rexec:45 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: rexec:113 setting Subsystem sftp
/usr/libexec/sftp-server
debug1: sshd version OpenSSH_9.2, OpenSSL 1.1.1w 11 Sep 2023
debug1: private host key #0: ecdsa-sha2-nistp256
SHA256:7OUDaY7vmsaJPDkqGWPmdiw5kjY4bVSwCd94nJqT7/o
debug1: private host key #1: ssh-ed25519
SHA256:CuPO+bnbHMCkaNEybTHeYSjdNpiNdAlntO9gh0V9lxs
debug1: private host key #2: ssh-rsa
SHA256:ZYZLLhbWdOMFKDGw3pcn954Wz6RhwtDoI5WjJsZpXhk
debug1: inetd sockets after dupping: 3, 3
debug3: process_channel_timeouts: setting 0 timeouts
debug3: channel_clear_timeouts: clearing
Connection from 192.168.124.214 port 57930 on 192.168.124.214 port
55222 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_9.2
debug1: Remote protocol version 2.0, remote software version
OpenSSH_8.4p1 Raspbian-5+deb11u2
debug1: compat_banner: match: OpenSSH_8.4p1 Raspbian-5+deb11u2 pat
OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
[Detaching after fork from child process 32308]
debug2: Network child is on pid 32308
debug3: preauth child monitor started
debug3: privsep user:group 107:65534 [preauth]
debug1: permanently_set_uid: 107/65534 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program
[preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive: entering

Dumping `/proc/$parentpid/status` just before the `read` failure shows:

Seccomp: 0
Seccomp_filters: 0

Dumping `/proc/$childpid/status` just before the `read` failure shows:

Seccomp: 3
Seccomp_filters: 1

File descriptor 5 of the parent process is a STREAM (according to
`lsof`)

$ lsof -p 11715
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
NAME
sshd 11715 root cwd DIR 179,2 4096 2
/
sshd 11715 root rtd DIR 179,2 4096 2
/
sshd 11715 root txt REG 179,2 1318404 72912
/root/Projects/openssh-9.2p1-WIP/sshd
sshd 11715 root mem REG 179,2 42628 2737
/lib/arm-linux-gnueabihf/libnss_files-2.31.so
sshd 11715 root mem REG 179,2 116324 4913
/lib/arm-linux-gnueabihf/libgcc_s.so.1
sshd 11715 root mem REG 179,2 137364 2748
/lib/arm-linux-gnueabihf/libpthread-2.31.so
sshd 11715 root mem REG 179,2 13864 2668
/lib/arm-linux-gnueabihf/libdl-2.31.so
sshd 11715 root mem REG 179,2 1315688 2667
/lib/arm-linux-gnueabihf/libc-2.31.so
sshd 11715 root mem REG 179,2 95880 11965
/lib/arm-linux-gnueabihf/libz.so.1.2.11
sshd 11715 root mem REG 179,2 2150824 11138
/usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1
sshd 11715 root mem REG 179,2 9796 2753
/lib/arm-linux-gnueabihf/libutil-2.31.so
sshd 11715 root mem REG 179,2 210340 1499
/lib/arm-linux-gnueabihf/libcrypt.so.1.1.0
sshd 11715 root mem REG 179,2 17708 14672
/usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so
sshd 11715 root mem REG 179,2 146888 2655
/lib/arm-linux-gnueabihf/ld-2.31.so
sshd 11715 root 0u CHR 1,3 0t0 5
/dev/null
sshd 11715 root 1u CHR 1,3 0t0 5
/dev/null
sshd 11715 root 2u CHR 136,4 0t0 7
/dev/pts/4
sshd 11715 root 3u IPv4 8165255 0t0 TCP
localhost:55522->localhost:48024 (ESTABLISHED)
sshd 11715 root 5u unix 0x00000000c618f590 0t0 8165266
type=STREAM

### Other miscellaneous observations:

* the child process quickly becomes "defunct"

Oddly, I can see that a child process is created by debug-printing PIDs
at certain points. e.g. a debug log message prints

debug2: Network child is on pid 11719

But later, just before the failed `read`, that child process is in a
"defunct" state. e.g. command `ps -ef` shows

$ ps -ef
...
sshd: [accepted]
[sshd] <defunct>

I suspect the child process is immediately dying and the later parent
process `read` then fails.

* Years ago on this same system, I locally built 8.4p1, 8.6p1, 9.0p1,
that have run just fine.
8.4p1 was built Feb 2021
8.6p1 was built Jun 2022
9.0p1 was built Jun 2022

Yet, I downloaded those same old versions today and they failed. Each
hit the same child process abort.

* I verified the address of `f` is function `read`
(https://github.com/openssh/openssh-portable/blob/V_9_2_P1/atomicio.c#L66)
with code snippet:

if (f == read) {
debug3("read(%d, '%s', %u); (errno=%u)", fd, s + pos, n - pos,
errno);
} else {
debug3("(f=%p) (%d, '%s', %u); (errno=%u)", f, fd, s + pos, n
- pos, errno);
}

> Could you add try adding a similar printf+getpid+exit

* verified within the `sshd` process, `__NR_epoll_create1 = 357` and
`__NR_getpid = 20` via `debug3` prints, e.g. code

debug3("__NR_getpid = %d", __NR_getpid);
debug3("__NR_epoll_create1 = %d", __NR_epoll_create1);
int _pid = getpid();
debug3("getpid() = %d", _pid);

* built with `./configure --with-sandbox=no` and it runs okay (no child
process aborts)

* other sandboxes failed to compile due to missing headers or kernel
capabilities (and I didn't feel like chasing these down)
* systrace
* rlimit
* capsicum

* Various `fcntl` `GET` checks of file descriptor 5. `errno` was set to
`0` before each call to `fcntl`.

fcntl(5, F_GETFD) returned 1 (0x00000001) (errno=0)
fcntl(5, F_GETOWN_EX) returned 0 (0x00000000) (errno=0)
owner.type=0, owner.pid=0
fcntl(5, F_GETOWN) returned 0 (0x00000000) (errno=0)
fcntl(5, F_GETPIPE_SZ) returned -1 (0xffffffff) (errno=9)
fcntl(5, F_GET_SEALS) returned -1 (0xffffffff) (errno=22)
fcntl(5, F_GETLEASE ) returned 2 (0x00000002) (errno=0)

* for posterity, if anyone else can repro this,
then manually add this code in `atomicio.c` function `atomicio6` to
cause a GDB break:

if (fd == 5 && n == 4 && pos == 0 && errno == 32) {
raise(SIGINT);
}

Those are the happenstance values before the failed `read` call.
Add the prior snippet just before code:

res = (f) (fd, s + pos, n - pos);

gdb command:

$ gdb --args "$(realpath .)/sshd" -ddddd -f sshd_config

### Summary Thoughts

> So perhaps the problem here is that either it's picking up 32bit vs 64bit headers, or that the binary is some kind of 32bit compatibility mode but the sandbox is expecting the 64bit syscalls.

Looking at the above `grep -r -Ee '__NR_getpid|__NR_epoll_create1'`,
maybe the resident compiled libc
used header `/usr/include/asm-generic/unistd.h` where `#define
__NR_epoll_create1 20`,
and openssh-server builds is uses header
`/usr/include/arm-linux-gnueabihf/asm/unistd-eabi.h`
where `#define __NR_getpid (__NR_SYSCALL_BASE + 20)` (or vice-versa).

Looking at the files

$ l /usr/include/arm-linux-gnueabihf/asm/unistd-eabi.h
-rw-r--r-- 1 root root 19938 Apr 5 2023
/usr/include/arm-linux-gnueabihf/asm/unistd-eabi.h

$ l /usr/include/asm-generic/unistd.h
-rw-r--r-- 1 root root 31480 Apr 5 2023
/usr/include/asm-generic/unistd.h

Looking at the compiled `libc`

$ find /usr -name 'libc.so'
/usr/lib/arm-linux-gnueabihf/libc.so

$ ls -l /usr/lib/arm-linux-gnueabihf/libc.so
-rw-r--r-- 1 root root 289 Oct 3 12:55
/usr/lib/arm-linux-gnueabihf/libc.so

So maybe my include or library pathing is was reconfigured (messed up)
some time in October 2023 (oh man, how screwed am I?)

### in the meantime ...

I have a workaround using `--with-sandbox=no`.

If you'd like me to try something else then please let me know.
Otherwise, I've spent a fair amount of time in this rabbit hole and
need to get back to other things (i.e. other rabbit holes ????).


-James Moon
(https://github.com/jtmoon79/)
(https://twitter.com/jtmoon1979/)

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #27 from Damien Miller <djm@mindrot.org> ---
Could you please provide this information so we can try to replicate it
ourselves:

(In reply to Darren Tucker from comment #11)
> I'd like to reproduce this locally. Could you please attach
> /etc/os-release and the output of "dpkg -l" from the affected device?

Also my request from comment #25:

> Could you add try adding a similar printf+getpid+exit sequence to
> (say) the start of ssh-keygen.c, rerunning make and repeating the
> test using the resultant binary? You probably only need to do this
> on the rpi4

Also, if you can catch the sandbox-violation in gdb, getting a
disassembly of instructions around the violation would be instructive.

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #29 from JM <jtm.moon.forum.user+mindrot@gmail.com> ---
> Also, if you can catch the sandbox-violation in gdb, getting a disassembly of
> instructions around the violation would be instructive.

tl;dr reviewing disassembly of a `getpid` call that violates the
sandbox shows
value `20` is the SYSCALL value used.
That SYSCALL value is also in a `SC_ALLOW` BPF filter.

### DEBUG SESSION ###

#### Debug Steps ####

1. modify `Makefile` to build with debugging symbols:
CFLAGS=-DDSANDBOX_SECCOMP_FILTER_DEBUG -ggdb -O0 ...
LDFLAGS=-ggdb -O0 ...
2. In `sandbox-seccomp-filter.c`
a. before the call to `prctl(PR_SET_SECCOMP, ...)`, I added a
`sleep(20)` and print `getpid()`

https://github.com/openssh/openssh-portable/blob/V_9_2_P1/sandbox-seccomp-filter.c#L438
This prints the child process PID, and allows time to manually
attach the `gdb` to the new child process.
b. Then added call to `raise(SIGINT)` so gdb will break just before
`prctl(PR_SET_SECCOMP, ...)`.
c. After the call to `prctl(PR_SET_SECCOMP, ...)` I added a call to
`getpid()`

https://github.com/openssh/openssh-portable/blob/V_9_2_P1/sandbox-seccomp-filter.c#L445
which will predictably invoke the sandbox violation.
3. run the `sshd` program
a. modify `sshd_config` to use port 55522
b. build and run
(set -eux; make clean; make sshd; "$(realpath .)/sshd" -ddd
-f ./sshd_config)
6. run the client ssh to invoke the error
ssh -v root@localhost -p 55522
7. when the `sshd` process sleeps for 20, attach `gdb`
gdb "--se=$(realpath ./sshd)" --pid=12345
8. in `gdb`
a. `continue`
it will break at `getpid` just after the `prctl` call
b. `stepi` into the call to `getpid`

See code in file `bpf-filters-dumped.txt`

#### Debug Result ####

Here is that run of `gdb` controlling the child process.

The child process terminates with `SIGSYS, Bad system call` after call
to `getpid`
when `getpid` is called *after* the sandbox filters are set by call
`prctl(PR_SET_SECCOMP, ...)`
https://github.com/openssh/openssh-portable/blob/V_9_2_P1/sandbox-seccomp-filter.c#L439

$ gdb "--se=$(realpath ./sshd)" --pid=16588
...
(gdb) step
Single stepping until exit from function raise,
which has no line number information.
ssh_sandbox_child (box=0x1506010) at sandbox-seccomp-filter.c:648
648 if (prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER,
&preauth_program) == -1) {
(gdb) step
654 else if (nnp_failed) {
(gdb) step
662 pid_t pid_2 = getpid();
(gdb) stepi
0x008d62e4 in getpid@plt ()
(gdb) stepi
0x008d62e8 in getpid@plt ()
(gdb) stepi
0x008d62ec in getpid@plt ()
(gdb) stepi
0xf7c04800 in getpid () from /lib/arm-linux-gnueabihf/libc.so.6
(gdb) stepi
0xf7c04804 in getpid () from /lib/arm-linux-gnueabihf/libc.so.6
(gdb) disassemble
Dump of assembler code for function getpid:
0xf7c04800 <+0>: push {r7} ; (str r7, [sp,
#-4]!)
=> 0xf7c04804 <+4>: mov r7, #20
0xf7c04808 <+8>: svc 0x00000000
0xf7c0480c <+12>: pop {r7} ; (ldr r7, [sp],
#4)
0xf7c04810 <+16>: bx lr
End of assembler dump.
(gdb) stepi
0xf7c04808 in getpid () from /lib/arm-linux-gnueabihf/libc.so.6
(gdb) disassemble
Dump of assembler code for function getpid:
0xf7c04800 <+0>: push {r7} ; (str r7, [sp,
#-4]!)
0xf7c04804 <+4>: mov r7, #20
=> 0xf7c04808 <+8>: svc 0x00000000
0xf7c0480c <+12>: pop {r7} ; (ldr r7, [sp],
#4)
0xf7c04810 <+16>: bx lr
End of assembler dump.
(gdb) stepi

Program terminated with signal SIGSYS, Bad system call.
The program no longer exists.

See file `gdb-output.txt` for the full output.

### DUMP OF BPF FILTERS ###

Before the call to `prctl(PR_SET_SECCOMP, ...)`, I added a print of
`__NR_getpid`
https://github.com/openssh/openssh-portable/blob/V_9_2_P1/sandbox-seccomp-filter.c#L438

#ifdef __NR_getpgid
debug3("SC_ALLOW(__NR_getpgid=%d),", __NR_getpgid);
#endif

That code snippet is taken from the creation of the BPF filter defined
at compile-time.


https://github.com/openssh/openssh-portable/blob/V_9_2_P1/sandbox-seccomp-filter.c#L229

This prints `__NR_getpid=20` which means it is defined and BPF allowed.

debug3: SC_ALLOW(__NR_getpid=20), [preauth]

See file `bpf-filters-dumped.txt`.

--
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #30 from JM <jtm.moon.forum.user+mindrot@gmail.com> ---
Created attachment 3779
--> https://bugzilla.mindrot.org/attachment.cgi?id=3779&action=edit
gdp-output.txt

gdb output before the errorring call to getpid

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs
[Bug 3639] server thread aborts during client login after receiving SSH2_MSG_KEXINIT due to ssh_sandbox_violation [ In reply to ]
https://bugzilla.mindrot.org/show_bug.cgi?id=3639

--- Comment #31 from JM <jtm.moon.forum.user+mindrot@gmail.com> ---
Created attachment 3780
--> https://bugzilla.mindrot.org/attachment.cgi?id=3780&action=edit
bpf-filters-dumped.txt

printing the BPF filters at runtime

--
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.
_______________________________________________
openssh-bugs mailing list
openssh-bugs@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-bugs