Mailing List Archive

Subsystem sftp invoked even though forced command created
Folks,

I'm curious if the documented behavior of portable OpenSSH (specifically Linux) may be at odds with the actual behavior I have seen in my experiments. Here is the background:

I manage an application which collects data from a client script (Korn shell) which runs on Unix and Linux servers across the entire enterprise. The client communicates with a Linux server (currently running RHEL 7.9, reporting OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017) using a technique we call SSH Pass-Through and described in an article in ;login: (Secure Automated File Transfer, vol. 30, no. 4, August 2005, by Mark McCullough who was also the original author of the client). The technique is predicated on using a forced command, which we specify per key with the "command=" directive in the authorized_keys file. (The user ID we use performs several functions, so we cannot use ForceCommand in configuration.)

In theory, any SSH connection made to the account with the corresponding private key should be forced to run the command provided, which accepts a fully-qualified domain name, environment, and stream of XML data on standard input and stores the data in a file with a name reflecting the FQDN and environment along with a date stamp. In all documentation I have to hand, the forced command is applied to all sessions including interactive, non-interactive, and *subsystem calls*.

An authorized penetration tester brought to my attention that the private key embedded in the application can be extracted and used to launch a WinSCP session against the user ID which the client uses to send the data to the server. This session has the same access to all files on the server which the original user ID does, including the ability to traverse directories. This should not happen if the documentation is correct.

To test the theory, I've slightly altered the forced command to output log data to indicate whether the forced command is even executed, and if so, what command line it was passed from the client (or WinSCP). The added code is as follows:

LOGFILE=/tmp/name-of-file.log
if [[ $SSH_ORIGINAL_COMMAND ]]; then
print "Command: $SSH_ORIGINAL_COMMAND" >> $LOGFILE
else
print "No SSH_ORIGINAL COMMAND set" >> $LOGFILE
fi

I ran the client as is and received the following entry in the log:
Command: 2>/dev/null

I then set a flag in the client which removes the redirection to /dev/null and received:
No SSH_ORIGINAL COMMAND set

Finally, I connected via WinSCP as instructed by the pen tester and received no further data in the log. This indicates to me that the sftp subsystem does not even execute the forced command.

The subsystem configuration in sshd_config is
# override default of no subsystems
Subsystem sftp /usr/libexec/openssh/sftp-server

I would almost expect this sort of behavior with the internal-sftp subsystem, but not with an external executable. Can anyone shed some light on this situation?

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Subsystem sftp invoked even though forced command created [ In reply to ]
On 30.06.23 00:06, MCMANUS, MICHAEL P wrote:
> An authorized penetration tester brought to my attention that the private
> key embedded in the application can be extracted and used to launch a
> WinSCP session against the user ID which the client uses to send the data
> to the server.

As it happens, I have a system using dedicated keypairs and forced
commands configured for them to extract survey data from CentOS 7 boxes,
so let me try that ...

> $ ssh -t -q autoquest@bongo -p 29056 -i .ssh/id_uptime_ed25519
> 1688110066
> 1684949224
> 685215
> 0

> $ sftp -P 29056 -i .ssh/id_uptime_ed25519 -q autoquest@bongo
> Received message too long 825636920

Hm. Some specific quirk of WinSCP, maybe ... ?

[grabs Win10 box] [updates WinSCP to 6.1.1] [adds keypair to both ends]

... gets me an error (-> screenshot) suggesting that it received the
output from the forced command, and logs that the sshd has indeed run
the forced command. Sorry, cannot confirm so far ...

> I ran the client as is and received the following entry in the log:
> Command: 2>/dev/null

That's a weird, I'd even say nonfunctional, remote command, and makes me
suspect that your ssh command has a syntax problem ... ?

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
Re: Subsystem sftp invoked even though forced command created [ In reply to ]
On Thu, 29 Jun 2023, MCMANUS, MICHAEL P wrote:

> Folks,
>
> I'm curious if the documented behavior of portable OpenSSH (specifically Linux) may be at odds with the actual behavior I have seen in my experiments. Here is the background:

[snip]

It's very hard to figure out what is happening here without a debug log.

You can get one by stopping the listening sshd and running it manually
in debug mode, e.g. "/usr/sbin/sshd -ddd"

Separate traces from expected vs problematic behaviour would be most
helpful.

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Subsystem sftp invoked even though forced command created [ In reply to ]
On 30/06/2023 09:56, Damien Miller wrote:
> It's very hard to figure out what is happening here without a debug log.
>
> You can get one by stopping the listening sshd and running it manually
> in debug mode, e.g. "/usr/sbin/sshd -ddd"

Or starting one in debug mode on a different port, e.g. "-p99 -ddd"

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
RE: Subsystem sftp invoked even though forced command created [ In reply to ]
The value of $SSH_ORIGINAL_COMMAND (which is what is being logged here) reflects the fact that no command is present on the client's ssh command line. The script relies on the forced command (which is supposed to ignore the client command line otherwise and pass the value of $SSH_ORIGINAL_COMMAND on to the forced command script).

The actual command is similar to the following (parameters inserted to protect the source):
(print ${FQDN} ; print ${Environment} ; cat ${OutFileXML}) | \
ssh -Ti ${EmbeddedPrivateKey} \
-o HostKeyAlias="${Alias}" \
-o GlobalKnownHostsFile="${EmbeddedKnownHosts}" \
-o UserKnownHostsFile="${ClientSpecificKnownHosts}" \
-o StrictHostKeyChecking="yes" \
-o CheckHostIP="no" \
-o NumberOfPasswordPrompts=0 \
${User}@${Host} 2>/dev/null

-----Original Message-----
From: Jochen Bern <Jochen.Bern@binect.de>
Sent: Friday, June 30, 2023 1:12 AM
To: openssh-unix-dev@mindrot.org
Cc: MCMANUS, MICHAEL P <mm1072@att.com>
Subject: Re: Subsystem sftp invoked even though forced command created

On 30.06.23 00:06, MCMANUS, MICHAEL P wrote:
> An authorized penetration tester brought to my attention that the private
> key embedded in the application can be extracted and used to launch a
> WinSCP session against the user ID which the client uses to send the data
> to the server.

As it happens, I have a system using dedicated keypairs and forced
commands configured for them to extract survey data from CentOS 7 boxes,
so let me try that ...

> $ ssh -t -q autoquest@bongo -p 29056 -i .ssh/id_uptime_ed25519
> 1688110066
> 1684949224
> 685215
> 0

> $ sftp -P 29056 -i .ssh/id_uptime_ed25519 -q autoquest@bongo
> Received message too long 825636920

Hm. Some specific quirk of WinSCP, maybe ... ?

[grabs Win10 box] [updates WinSCP to 6.1.1] [adds keypair to both ends]

... gets me an error (-> screenshot) suggesting that it received the
output from the forced command, and logs that the sshd has indeed run
the forced command. Sorry, cannot confirm so far ...

> I ran the client as is and received the following entry in the log:
> Command: 2>/dev/null

That's a weird, I'd even say nonfunctional, remote command, and makes me
suspect that your ssh command has a syntax problem ... ?

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
RE: Subsystem sftp invoked even though forced command created [ In reply to ]
OK, I have obtained debug logs. The first one is from running the client; the second one is from WinSCP. The client has multiple keys embedded so the client selected an ECDSA key in place of the RSA key. However, the pen tester used the RSA key to demonstrate the issue so that is what was exported for the use of WinSCP.

Mike McManus
Principal – Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com

-----Original Message-----
From: Brian Candler <b.candler@pobox.com>
Sent: Friday, June 30, 2023 2:21 AM
To: MCMANUS, MICHAEL P <mm1072@att.com>
Cc: openssh-unix-dev@mindrot.org
Subject: Re: Subsystem sftp invoked even though forced command created

On 30/06/2023 09:56, Damien Miller wrote:
> It's very hard to figure out what is happening here without a debug log.
>
> You can get one by stopping the listening sshd and running it manually
> in debug mode, e.g. "/usr/sbin/sshd -ddd"

Or starting one in debug mode on a different port, e.g. "-p99 -ddd"
Re: Subsystem sftp invoked even though forced command created [ In reply to ]
On 30.06.23 17:56, MCMANUS, MICHAEL P wrote:
> The actual command is similar to the following (parameters inserted to protect the source):
> (print ${FQDN} ; print ${Environment} ; cat ${OutFileXML}) | \
> ssh -Ti ${EmbeddedPrivateKey} \
> -o HostKeyAlias="${Alias}" \
> -o GlobalKnownHostsFile="${EmbeddedKnownHosts}" \
> -o UserKnownHostsFile="${ClientSpecificKnownHosts}" \
> -o StrictHostKeyChecking="yes" \
> -o CheckHostIP="no" \
> -o NumberOfPasswordPrompts=0 \
> ${User}@${Host} 2>/dev/null

Then whatever executes this command line does *not* understand (and eat)
the "2>/dev/null" like shells of the Bourne family should, hence it
winding up in the server-side $SSH_ORIGINAL_COMMAND ...

> debug1: server_input_channel_req: channel 0 request subsystem reply 1
> debug1: session_by_channel: session 0 channel 0
> debug1: session_input_channel_req: session 0 req subsystem
> debug2: subsystem request for sftp by user m61586
> debug1: subsystem: exec() /usr/libexec/openssh/sftp-server
> Starting session: forced-command (key-option) '/opt/app/workload/secgov/opt/sact-central/bin/receive.ksh' for m61586 from 135.99.45.25 port 49734 id 0

Well, I'll be ... Looks like the server side recognizes the
forced-command setup, but honors the SFTP subsystem request by exec()ing
that nonetheless ... !

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
Re: Subsystem sftp invoked even though forced command created [ In reply to ]
On Mon, 3 Jul 2023, Jochen Bern wrote:

> On 30.06.23 17:56, MCMANUS, MICHAEL P wrote:
> > The actual command is similar to the following (parameters inserted to
> > protect the source):
> > (print ${FQDN} ; print ${Environment} ; cat ${OutFileXML}) | \
> > ssh -Ti ${EmbeddedPrivateKey} \
> > -o HostKeyAlias="${Alias}" \
> > -o GlobalKnownHostsFile="${EmbeddedKnownHosts}" \
> > -o UserKnownHostsFile="${ClientSpecificKnownHosts}" \
> > -o StrictHostKeyChecking="yes" \
> > -o CheckHostIP="no" \
> > -o NumberOfPasswordPrompts=0 \
> > ${User}@${Host} 2>/dev/null
>
> Then whatever executes this command line does *not* understand (and eat) the
> "2>/dev/null" like shells of the Bourne family should, hence it winding up in
> the server-side $SSH_ORIGINAL_COMMAND ...
>
> > debug1: server_input_channel_req: channel 0 request subsystem reply 1
> > debug1: session_by_channel: session 0 channel 0
> > debug1: session_input_channel_req: session 0 req subsystem
> > debug2: subsystem request for sftp by user m61586
> > debug1: subsystem: exec() /usr/libexec/openssh/sftp-server
> > Starting session: forced-command (key-option)
> > '/opt/app/workload/secgov/opt/sact-central/bin/receive.ksh' for m61586 from
> > 135.99.45.25 port 49734 id 0
>
> Well, I'll be ... Looks like the server side recognizes the forced-command
> setup, but honors the SFTP subsystem request by exec()ing that nonetheless ...

no, that's the log message being misleading. At the point of the
"Starting session: forced-command" in session.c:do_exec() the command
variable contains the requested forced command and that is what is
actually executed.

https://github.com/openssh/openssh-portable/blob/V_7_4_P1/session.c#L623-L663

The "subsystem: exec()" comes earlier from
session.c:session_subsystem_req(), which is recording (badly) which
subsystem was requested, but this is before the forced command is applied
in do_exec().

https://github.com/openssh/openssh-portable/blob/V_7_4_P1/session.c#L1938-L1944

Further evidence that this is the case is the audit calls that log the
command being executed:

debug3: mm_audit_run_command entering command
/opt/app/workload/secgov/opt/sact-central/bin/receive.ksh
debug3: mm_audit_end_command entering command
/opt/app/workload/secgov/opt/sact-central/bin/receive.ksh

So the command appears to have been correctly overridden and nothing in
the debug logs explains the behaviour you're reporting.

Some possibilities:

1. the receive.ksh script is faulty in some way that causes it to invoke
sftp-server
2. some changes made by Redhat in the binaries they provided, not present
in the OpenSSH source we release have broken forced commands.

#2 could be excluded by reproducing the problem using a sshd built from
source, without redhat's patches.

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Subsystem sftp invoked even though forced command created [ In reply to ]
On 05.07.23 02:50, Damien Miller wrote:
> Some possibilities:
> 1. the receive.ksh script is faulty in some way that causes it to invoke
> sftp-server

How would the script even *know* that the client requested the SFTP
subsystem? Is a subsystem's executable/path, supposedly internally
overwritten with the forced command at that point, exposed through
$SSH_ORIGINAL_COMMAND ?

(As a quick preliminary check, I'd suggest doing a "ps auwwwx --forest"
on the server while WinSCP has a "hacked" session open. If the
sftp-server process turns out to be a child of the script, bingo. If
not, the script could still be the culprit, but then we'd know that it
must "exec" the sftp-server or somesuch, rather than calling it
"normally" as a subprocess.)

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
RE: Subsystem sftp invoked even though forced command created [ In reply to ]
It appears the forced command either does not run or runs to completion and exits immediately, as there is no process named "receive.ksh" in the process tree.

The sftp-server process is an immediate child of the privilege-separation sshd process:
root 1157 0.0 0.1 94556 5804 ? Ss Jun07 0:00 /usr/sbin/sshd -D
root 3933778 0.0 0.2 155624 9732 ? Ss 10:34 0:00 \_ sshd: mm1072 [priv]
mm1072 3933794 0.0 0.1 155624 5564 ? S 10:34 0:00 | \_ sshd: mm1072@pts/0
mm1072 3933795 0.0 0.1 25428 5252 pts/0 Ss 10:34 0:00 | \_ -bash
mm1072 3934980 0.0 0.1 59200 4636 pts/0 R+ 10:57 0:00 | \_ ps auwwwx --forest
root 3934958 0.1 0.2 155628 10568 ? Ss 10:56 0:00 \_ sshd: m61586 [priv]
m61586 3934972 0.0 0.1 155628 5576 ? S 10:56 0:00 \_ sshd: m61586@notty
m61586 3934973 0.0 0.1 47280 5228 ? Ss 10:56 0:00 \_ /usr/libexec/openssh/sftp-server

Mike McManus
Principal – Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com


-----Original Message-----
From: openssh-unix-dev <openssh-unix-dev-bounces+mm1072=att.com@mindrot.org> On Behalf Of Jochen Bern
Sent: Wednesday, July 5, 2023 1:52 AM
To: openssh-unix-dev@mindrot.org
Subject: Re: Subsystem sftp invoked even though forced command created

On 05.07.23 02:50, Damien Miller wrote:
> Some possibilities:
> 1. the receive.ksh script is faulty in some way that causes it to invoke
> sftp-server

How would the script even *know* that the client requested the SFTP
subsystem? Is a subsystem's executable/path, supposedly internally
overwritten with the forced command at that point, exposed through
$SSH_ORIGINAL_COMMAND ?

(As a quick preliminary check, I'd suggest doing a "ps auwwwx --forest"
on the server while WinSCP has a "hacked" session open. If the
sftp-server process turns out to be a child of the script, bingo. If
not, the script could still be the culprit, but then we'd know that it
must "exec" the sftp-server or somesuch, rather than calling it
"normally" as a subprocess.)

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Subsystem sftp invoked even though forced command created [ In reply to ]
On Wed, 5 Jul 2023, Damien Miller wrote:

> Some possibilities:
>
> 1. the receive.ksh script is faulty in some way that causes it to invoke
> sftp-server
> 2. some changes made by Redhat in the binaries they provided, not present
> in the OpenSSH source we release have broken forced commands.
>
> #2 could be excluded by reproducing the problem using a sshd built from
> source, without redhat's patches.

FWIW, I've rebuilt openssh-7.4 from source and attempted a sftp connection
with a ForceCommand set in sshd_config. The sftp session fails as expected:

[djm@djm ~]$ sftp -oPort=2222 ::1
Authenticated to ::1 ([::1]:2222) using "publickey".
Received message too long 1482184792
Ensure the remote shell produces no output for non-interactive sessions.
Transferred: sent 2260, received 1520 bytes, in 0.0 seconds
Bytes per second: sent 614738.3, received 413452.3

On the server side:

debug2: subsystem request for sftp by user djm
debug1: subsystem: exec() /usr/libexec/sftp-server
Starting session: forced-command (config) '/tmp/xxx' for djm from ::1 port 32870 id 0


_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: RE: Subsystem sftp invoked even though forced command created [ In reply to ]
On 05.07.23 18:01, MCMANUS, MICHAEL P wrote:
> It appears the forced command either does not run or runs to completion
> and exits immediately, as there is no process named "receive.ksh" in
> the process tree.

FWIW, two cents of mine:

-- The script *exiting* should *not* prompt sshd to execute the
requested subsystem "as a second thought", or else it'd happen all over
the place.

-- The process' cmdline would state the shell executing the script (ksh,
I suppose?) rather than the script file.

In the meantime, I received an (off-list) e-mail pointing out that your
script obviously accepts input from stdin (note the "-T" given to ssh,
so no tty):

>> The actual command is similar to the following (parameters inserted to protect the source):
>> (print ${FQDN} ; print ${Environment} ; cat ${OutFileXML}) | \
>> ssh -Ti ${EmbeddedPrivateKey} ...

and that it's conceivable that WinSCP might send a command line
executing sftp-server, just in case the server provides it with a login
shell instead of calling the SFTP subsystem directly; Hence the theory
that the script has some command injection vulnerability.

Does the exploit still work when you change the authorized_keys from
command="/.../receive.ksh"
to, e.g.,
command="/bin/ksh -c '/.../receive.ksh </dev/null'"
to suppress the client's input?

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
RE: RE: Subsystem sftp invoked even though forced command created [ In reply to ]
The forced command actively uses the client's input to feed data to the server (it constructs a file name from the information supplied by the client). So changing the forced command as stated will break the application. I would need to create a test bed to simulate the listener rather than use the server as is, where is. That may produce false or misleading results.

A pseudocode description of the forced command follows:
Read one line from stdin and store to $FQDN.
Read one line from stdin and store to $ENVIRONMENT.
Construct a temporary and a permanent file name using the system date, process ID, $FQDN, and $ENVIRONMENT.
Slurp up (dd) all remaining data from stdin and store in the temporary file name constructed in the previous step.
Perform some textual cleanup on the temporary file.
Save the cleaned-up file to the permanent file name.
Emit a success message to stdout.

Obviously there is also error handling, which emits error messages to stdout. But WinSCP is giving no indication that anything has been sent to it during the session setup.

Oddly enough, the same behavior occurs when the embedded key is used to launch an interactive sftp session from the host running the legitimate client:
# sftp -i ${embeddedKey} ${user}@${host}
<Standard warning from /etc/issue.net>
Connected to ${host}.
README collectors receive-data.ksh tmp
So we can probably write off any idiosyncrasies of WinSCP and work only with OpenSSH. Note there is no output from the script whatsoever.

Mike McManus
Principal – Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com

-----Original Message-----
From: Jochen Bern <Jochen.Bern@binect.de>
Sent: Thursday, July 6, 2023 4:20 AM
To: openssh-unix-dev@mindrot.org
Cc: MCMANUS, MICHAEL P <mm1072@att.com>
Subject: Re: RE: Subsystem sftp invoked even though forced command created

On 05.07.23 18:01, MCMANUS, MICHAEL P wrote:
> It appears the forced command either does not run or runs to completion
> and exits immediately, as there is no process named "receive.ksh" in
> the process tree.

FWIW, two cents of mine:

-- The script *exiting* should *not* prompt sshd to execute the
requested subsystem "as a second thought", or else it'd happen all over
the place.

-- The process' cmdline would state the shell executing the script (ksh,
I suppose?) rather than the script file.

In the meantime, I received an (off-list) e-mail pointing out that your
script obviously accepts input from stdin (note the "-T" given to ssh,
so no tty):

>> The actual command is similar to the following (parameters inserted to protect the source):
>> (print ${FQDN} ; print ${Environment} ; cat ${OutFileXML}) | \
>> ssh -Ti ${EmbeddedPrivateKey} ...

and that it's conceivable that WinSCP might send a command line
executing sftp-server, just in case the server provides it with a login
shell instead of calling the SFTP subsystem directly; Hence the theory
that the script has some command injection vulnerability.

Does the exploit still work when you change the authorized_keys from
command="/.../receive.ksh"
to, e.g.,
command="/bin/ksh -c '/.../receive.ksh </dev/null'"
to suppress the client's input?

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: RE: RE: Subsystem sftp invoked even though forced command created [ In reply to ]
On 06.07.23 23:37, MCMANUS, MICHAEL P wrote:> So changing the forced
command as stated will break the application. I
> would need to create a test bed to simulate the listener rather than
> use the server as is, where is. That may produce false or misleading
> results.
Since the forced command is tied to the specific keypair in the
authorized_keys, you could
-- test with a different keypair or
-- use an additional 'from="..."' option to split the entry between your
test client and the productive clients.

> Oddly enough, the same behavior occurs when the embedded key is used
> to launch an interactive sftp session from the host running the
> legitimate client:
>
> # sftp -i ${embeddedKey} ${user}@${host}
> <Standard warning from /etc/issue.net>
> Connected to ${host}.
> sftp> ls
> README collectors receive-data.ksh tmp
> sftp> ^D
> So we can probably write off any idiosyncrasies of WinSCP and work only
> with OpenSSH. Note there is no output from the script whatsoever.

In that case, let me repeat my quick test on one of our systems ... :

> [root ~]# cat /etc/centos-release
> CentOS Linux release 7.9.2009 (Core)
> [root ~]# rpm -q openssh
> openssh-7.4p1-22.el7_9.x86_64
> [root ~]# tail -1 ~autoquest/.ssh/authorized_keys | sed -e 's/AAA.*/.../'
> restrict,from="127.0.0.1",command="/bin/logger -t AutoHack" ssh-rsa ...
> [root ~]# ssh-keygen -l -f /home/autoquest/.ssh/authorized_keys | tail -1
> 4096 SHA256:NSG4SRm/sLQxX4Xc5lQiMc3Q9S5j0Vavp7gu+voAwhI CNG-000121900000-010098-01 (RSA)
> [root ~]# ssh-keygen -l -f /home/bongo/.ssh/*.pub
> 4096 SHA256:NSG4SRm/sLQxX4Xc5lQiMc3Q9S5j0Vavp7gu+voAwhI CNG-000121900000-010098-01 (RSA)
> [root ~]# su -l -s /bin/bash bongo
> [bongo ~]$ echo "foo bar baz" | sftp autoquest@127.0.0.1
[.... confirm host keypair, output of /etc/issue.net ... then it just
hangs ...]
> ^CKilled by signal 15.
> [bongo ~]$ exit
> logout
> [root ~]# journalctl -t AutoHack
> -- Logs begin at Thu 2023-06-22 11:07:33 CEST, end at Fri 2023-07-07 14:20:35 CEST. --
> Jul 07 14:19:35 cng-000121900000-010098-01 AutoHack[15837]:

... no SFTP login, but also no stdin being logged ...

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
RE: RE: RE: Subsystem sftp invoked even though forced command created [ In reply to ]
So I set up a fresh key to use for this test, and gave it similar parameters.

I wasn't aware of the "restrict" argument but the original authorized_keys entries had a long list of restrictions:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/.../receive.ksh" ssh-rsa ...

I did the same thing as I had done with the live key and got the same results (as expected).

It then occurred to me that the issue is probably on the server side, which did not change. So I set up the user and forced command on 3 lab servers:
OS OpenSSH version
Ubuntu 18.04 OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n 7 Dec 2017
Solaris 11 OpenSSH_8.4p1, OpenSSL 1.0.2zf 21 Jun 2022
HP-UX 11.31 OpenSSH_8.1p1+sftpfilecontrol-v1.3-hpn14v20, OpenSSL 1.1.1d 10 Sep 2019

I got the following results:
OS ssh sftp SSH_ORIGINAL_COMMAND
Ubuntu 18.04 Hung waiting for input, pressed ^C Obtained sftp prompt. Not logged
Solaris 11 Hung waiting for input, pressed ^C Hung waiting for input, pressed ^C Command: internal-sftp
HP-UX 11.31 Hung waiting for input, pressed ^C Hung waiting for input, pressed ^C Command: /opt/ssh/libexec/sftp-server

So the issue looks like it happens across Linux distributions, but not on non-Linux. Curiouser and curiouser.

I then changed the forced command to /usr/bin/logger and got similar results. I think the default priority for logs on our servers goes nowhere; next round I'll have to specify a priority that is actually logged.

Hope you have a great weekend, and thanks for your help so far.

--
Mike McManus
Principal – Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com

-----Original Message-----
From: Jochen Bern <Jochen.Bern@binect.de>
Sent: Friday, July 7, 2023 5:28 AM
To: openssh-unix-dev@mindrot.org
Cc: MCMANUS, MICHAEL P <mm1072@att.com>
Subject: Re: RE: RE: Subsystem sftp invoked even though forced command created

On 06.07.23 23:37, MCMANUS, MICHAEL P wrote:> So changing the forced
command as stated will break the application. I
> would need to create a test bed to simulate the listener rather than
> use the server as is, where is. That may produce false or misleading
> results.
Since the forced command is tied to the specific keypair in the
authorized_keys, you could
-- test with a different keypair or
-- use an additional 'from="..."' option to split the entry between your
test client and the productive clients.

> Oddly enough, the same behavior occurs when the embedded key is used
> to launch an interactive sftp session from the host running the
> legitimate client:
>
> # sftp -i ${embeddedKey} ${user}@${host}
> <Standard warning from /etc/issue.net>
> Connected to ${host}.
> sftp> ls
> README collectors receive-data.ksh tmp
> sftp> ^D
> So we can probably write off any idiosyncrasies of WinSCP and work only
> with OpenSSH. Note there is no output from the script whatsoever.

In that case, let me repeat my quick test on one of our systems ... :

> [root ~]# cat /etc/centos-release
> CentOS Linux release 7.9.2009 (Core)
> [root ~]# rpm -q openssh
> openssh-7.4p1-22.el7_9.x86_64
> [root ~]# tail -1 ~autoquest/.ssh/authorized_keys | sed -e 's/AAA.*/.../'
> restrict,from="127.0.0.1",command="/bin/logger -t AutoHack" ssh-rsa ...
> [root ~]# ssh-keygen -l -f /home/autoquest/.ssh/authorized_keys | tail -1
> 4096 SHA256:NSG4SRm/sLQxX4Xc5lQiMc3Q9S5j0Vavp7gu+voAwhI CNG-000121900000-010098-01 (RSA)
> [root ~]# ssh-keygen -l -f /home/bongo/.ssh/*.pub
> 4096 SHA256:NSG4SRm/sLQxX4Xc5lQiMc3Q9S5j0Vavp7gu+voAwhI CNG-000121900000-010098-01 (RSA)
> [root ~]# su -l -s /bin/bash bongo
> [bongo ~]$ echo "foo bar baz" | sftp autoquest@127.0.0.1
[.... confirm host keypair, output of /etc/issue.net ... then it just
hangs ...]
> ^CKilled by signal 15.
> [bongo ~]$ exit
> logout
> [root ~]# journalctl -t AutoHack
> -- Logs begin at Thu 2023-06-22 11:07:33 CEST, end at Fri 2023-07-07 14:20:35 CEST. --
> Jul 07 14:19:35 cng-000121900000-010098-01 AutoHack[15837]:

... no SFTP login, but also no stdin being logged ...

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
RE: Subsystem sftp invoked even though forced command created [ In reply to ]
This is a new branch of an old thread, made necessary because the email system here purges sent messages after a period of time so I can't reply to the last message in the thread. The operative portion of that last message (retrieved from the archives and dated July 3, 2023) follows:

/*****/
So I set up a fresh key to use for this test, and gave it similar parameters.

I wasn't aware of the "restrict" argument but the original authorized_keys entries had a long list of restrictions:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/.../receive.ksh" ssh-rsa ...

I did the same thing as I had done with the live key and got the same results (as expected).

It then occurred to me that the issue is probably on the server side, which did not change. So I set up the user and forced command on 3 lab servers:
OS OpenSSH version
Ubuntu 18.04 OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n 7 Dec 2017
Solaris 11 OpenSSH_8.4p1, OpenSSL 1.0.2zf 21 Jun 2022
HP-UX 11.31 OpenSSH_8.1p1+sftpfilecontrol-v1.3-hpn14v20, OpenSSL 1.1.1d 10 Sep 2019

I got the following results:
OS ssh sftp SSH_ORIGINAL_COMMAND
Ubuntu 18.04 Hung waiting for input, pressed ^C Obtained sftp prompt. Not logged
Solaris 11 Hung waiting for input, pressed ^C Hung waiting for input, pressed ^C Command: internal-sftp
HP-UX 11.31 Hung waiting for input, pressed ^C Hung waiting for input, pressed ^C Command: /opt/ssh/libexec/sftp-server

So the issue looks like it happens across Linux distributions, but not on non-Linux. Curiouser and curiouser.

I then changed the forced command to /usr/bin/logger and got similar results. I think the default priority for logs on our servers goes nowhere; next round I'll have to specify a priority that is actually logged.
/*****/

The conclusion I came to was that the Linux distributors are working with a modified version of the Berkeley sources (they have to, in order to back-port security fixes like they do). Therefore I downloaded OpenSSH 9.4p1 sources from the Berkeley depot and compiled them on a Rocky Linux 8 server.

Sadly, the problem remains. I noticed in the logs there were references to Red Hat specific configuration, but that was on the client side and not the server side. I'm beginning to think that there is something in the configuration of OpenSSH on Linux servers which prioritizes the sftp subsystem over the forced command, whether the binaries are downloaded from the OS distributor or built from sources.

Unless I can get some expert advice, I believe this kills off the use of SSH as a quick and secure transport for unattended file transfer and ingestion by applications, at least on Linux. It will have to be replaced by TLS or the Splunk Universal Agent. Pity, that.

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com

-----Original Message-----
From: openssh-unix-dev <openssh-unix-dev-bounces+mm1072=att.com@mindrot.org> On Behalf Of MCMANUS, MICHAEL P
Sent: Thursday, June 29, 2023 3:06 PM
To: openssh-unix-dev@mindrot.org
Subject: Subsystem sftp invoked even though forced command created

*** Security Advisory: This Message Originated Outside of AT&T ***.
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Folks,

I'm curious if the documented behavior of portable OpenSSH (specifically Linux) may be at odds with the actual behavior I have seen in my experiments. Here is the background:

I manage an application which collects data from a client script (Korn shell) which runs on Unix and Linux servers across the entire enterprise. The client communicates with a Linux server (currently running RHEL 7.9, reporting OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017) using a technique we call SSH Pass-Through and described in an article in ;login: (Secure Automated File Transfer, vol. 30, no. 4, August 2005, by Mark McCullough who was also the original author of the client). The technique is predicated on using a forced command, which we specify per key with the "command=" directive in the authorized_keys file. (The user ID we use performs several functions, so we cannot use ForceCommand in configuration.)

In theory, any SSH connection made to the account with the corresponding private key should be forced to run the command provided, which accepts a fully-qualified domain name, environment, and stream of XML data on standard input and stores the data in a file with a name reflecting the FQDN and environment along with a date stamp. In all documentation I have to hand, the forced command is applied to all sessions including interactive, non-interactive, and *subsystem calls*.

An authorized penetration tester brought to my attention that the private key embedded in the application can be extracted and used to launch a WinSCP session against the user ID which the client uses to send the data to the server. This session has the same access to all files on the server which the original user ID does, including the ability to traverse directories. This should not happen if the documentation is correct.

To test the theory, I've slightly altered the forced command to output log data to indicate whether the forced command is even executed, and if so, what command line it was passed from the client (or WinSCP). The added code is as follows:

LOGFILE=/tmp/name-of-file.log
if [[ $SSH_ORIGINAL_COMMAND ]]; then
print "Command: $SSH_ORIGINAL_COMMAND" >> $LOGFILE
else
print "No SSH_ORIGINAL COMMAND set" >> $LOGFILE
fi

I ran the client as is and received the following entry in the log:
Command: 2>/dev/null

I then set a flag in the client which removes the redirection to /dev/null and received:
No SSH_ORIGINAL COMMAND set

Finally, I connected via WinSCP as instructed by the pen tester and received no further data in the log. This indicates to me that the sftp subsystem does not even execute the forced command.

The subsystem configuration in sshd_config is
# override default of no subsystems
Subsystem sftp /usr/libexec/openssh/sftp-server

I would almost expect this sort of behavior with the internal-sftp subsystem, but not with an external executable. Can anyone shed some light on this situation?

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://urldefense.com/v3/__https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev__;!!BhdT!gAfqyQFKV_nTGhaz-PwBqHZat18A67O51MZnz1_jFaordOKkpGy9dSyhdHYW0dqNiT5SUd2Mh6M$
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
RE: Subsystem sftp invoked even though forced command created [ In reply to ]
Following up on this thread:

Splunk UF cannot be used to drive an external application as it uses a proprietary stream data format (over a TLS-protected channel), thus the only recipient of the data must be a Splunk server.

Three options have been presented to me:
* A standard HTTPS web server such as Apache, driven by curl on the client side
* Open Telemetry [https://opentelemetry.io/].
* Apache Nifi [https://nifi.apache.org/].

My suspicion is that there is a very deep, fundamental difference in one of the core libraries provided by Linux as compared to those provided by BSD or Sys V platforms (including Solaris, HP-UX, and AIX). None of the latter three exhibit the problem.

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20309 North Creek Pkwy, Bothell, WA 98011
michael.mcmanus@att.com

From: openssh-unix-dev <openssh-unix-dev-bounces+mm1072=att.com@mindrot.org> On Behalf Of MCMANUS, MICHAEL P
Sent: Tuesday, September 19, 2023 7:59 AM
To: openssh-unix-dev@mindrot.org
Subject: RE: Subsystem sftp invoked even though forced command created

This is a new branch of an old thread, made necessary because the email system here purges sent messages after a period of time so I can't reply to the last message in the thread. The operative portion of that last message (retrieved from the archives and dated July 3, 2023) follows:

/*****/
So I set up a fresh key to use for this test, and gave it similar parameters.

I wasn't aware of the "restrict" argument but the original authorized_keys entries had a long list of restrictions:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="/.../receive.ksh" ssh-rsa ...

I did the same thing as I had done with the live key and got the same results (as expected).

It then occurred to me that the issue is probably on the server side, which did not change. So I set up the user and forced command on 3 lab servers:
OS OpenSSH version
Ubuntu 18.04 OpenSSH_7.6p1 Ubuntu-4ubuntu0.7, OpenSSL 1.0.2n 7 Dec 2017
Solaris 11 OpenSSH_8.4p1, OpenSSL 1.0.2zf 21 Jun 2022
HP-UX 11.31 OpenSSH_8.1p1+sftpfilecontrol-v1.3-hpn14v20, OpenSSL 1.1.1d 10 Sep 2019

I got the following results:
OS ssh sftp SSH_ORIGINAL_COMMAND
Ubuntu 18.04 Hung waiting for input, pressed ^C Obtained sftp prompt. Not logged
Solaris 11 Hung waiting for input, pressed ^C Hung waiting for input, pressed ^C Command: internal-sftp
HP-UX 11.31 Hung waiting for input, pressed ^C Hung waiting for input, pressed ^C Command: /opt/ssh/libexec/sftp-server

So the issue looks like it happens across Linux distributions, but not on non-Linux. Curiouser and curiouser.

I then changed the forced command to /usr/bin/logger and got similar results. I think the default priority for logs on our servers goes nowhere; next round I'll have to specify a priority that is actually logged.
/*****/

The conclusion I came to was that the Linux distributors are working with a modified version of the Berkeley sources (they have to, in order to back-port security fixes like they do). Therefore I downloaded OpenSSH 9.4p1 sources from the Berkeley depot and compiled them on a Rocky Linux 8 server.

Sadly, the problem remains. I noticed in the logs there were references to Red Hat specific configuration, but that was on the client side and not the server side. I'm beginning to think that there is something in the configuration of OpenSSH on Linux servers which prioritizes the sftp subsystem over the forced command, whether the binaries are downloaded from the OS distributor or built from sources.

Unless I can get some expert advice, I believe this kills off the use of SSH as a quick and secure transport for unattended file transfer and ingestion by applications, at least on Linux. It will have to be replaced by TLS or the Splunk Universal Agent. Pity, that.

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
mailto:michael.mcmanus@att.com

-----Original Message-----
From: openssh-unix-dev <mailto:openssh-unix-dev-bounces+mm1072=att.com@mindrot.org> On Behalf Of MCMANUS, MICHAEL P
Sent: Thursday, June 29, 2023 3:06 PM
To: mailto:openssh-unix-dev@mindrot.org
Subject: Subsystem sftp invoked even though forced command created

*** Security Advisory: This Message Originated Outside of AT&T ***.
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Folks,

I'm curious if the documented behavior of portable OpenSSH (specifically Linux) may be at odds with the actual behavior I have seen in my experiments. Here is the background:

I manage an application which collects data from a client script (Korn shell) which runs on Unix and Linux servers across the entire enterprise. The client communicates with a Linux server (currently running RHEL 7.9, reporting OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017) using a technique we call SSH Pass-Through and described in an article in ;login: (Secure Automated File Transfer, vol. 30, no. 4, August 2005, by Mark McCullough who was also the original author of the client). The technique is predicated on using a forced command, which we specify per key with the "command=" directive in the authorized_keys file. (The user ID we use performs several functions, so we cannot use ForceCommand in configuration.)

In theory, any SSH connection made to the account with the corresponding private key should be forced to run the command provided, which accepts a fully-qualified domain name, environment, and stream of XML data on standard input and stores the data in a file with a name reflecting the FQDN and environment along with a date stamp. In all documentation I have to hand, the forced command is applied to all sessions including interactive, non-interactive, and *subsystem calls*.

An authorized penetration tester brought to my attention that the private key embedded in the application can be extracted and used to launch a WinSCP session against the user ID which the client uses to send the data to the server. This session has the same access to all files on the server which the original user ID does, including the ability to traverse directories. This should not happen if the documentation is correct.

To test the theory, I've slightly altered the forced command to output log data to indicate whether the forced command is even executed, and if so, what command line it was passed from the client (or WinSCP). The added code is as follows:

LOGFILE=/tmp/name-of-file.log
if [[ $SSH_ORIGINAL_COMMAND ]]; then
print "Command: $SSH_ORIGINAL_COMMAND" >> $LOGFILE
else
print "No SSH_ORIGINAL COMMAND set" >> $LOGFILE
fi

I ran the client as is and received the following entry in the log:
Command: 2>/dev/null

I then set a flag in the client which removes the redirection to /dev/null and received:
No SSH_ORIGINAL COMMAND set

Finally, I connected via WinSCP as instructed by the pen tester and received no further data in the log. This indicates to me that the sftp subsystem does not even execute the forced command.

The subsystem configuration in sshd_config is
# override default of no subsystems
Subsystem sftp /usr/libexec/openssh/sftp-server

I would almost expect this sort of behavior with the internal-sftp subsystem, but not with an external executable. Can anyone shed some light on this situation?

Mike McManus
Principal - Technology Security
GTO Security Governance Team - Unix
P: He/Him/His

AT&T Services, Inc.
20205 North Creek Pkwy, Bothell, WA 98011
mailto:michael.mcmanus@att.com

_______________________________________________
openssh-unix-dev mailing list
mailto:openssh-unix-dev@mindrot.org
https://urldefense.com/v3/__https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev__;!!BhdT!gAfqyQFKV_nTGhaz-PwBqHZat18A67O51MZnz1_jFaordOKkpGy9dSyhdHYW0dqNiT5SUd2Mh6M$
_______________________________________________
openssh-unix-dev mailing list
mailto:openssh-unix-dev@mindrot.org
https://urldefense.com/v3/__https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev__;!!BhdT!jB623mgetR7HfBahE8oix3S2IvTlS8gi8b9cEjavaF6wUCmbu2aiTNkro69I10HFjUhp8XhV14A$
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev