Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. OpenSSH>
  3. Dev
scp, sftp, and special characters in filenames
demiobenour at gmail
Nov 3, 2019, 8:59 AM
Post #1 of 9 (1400 views)
Permalink
I have not been able to get scp(1) to download a file with a newline
in its name. I know that scp(1) requires that remote filenames be
escaped for the shell, but that leads to protocol errors.

I am also unable to get sftp(1) to handle such files. It fails with
an “unterminated quoted argument” error, no matter how I try to
quote the newline. Furthermore, the SFTP CLI is not well-suited to
scripted transfers, as it requires creating batch files. If sftp(1)
could accept commands as command line arguments, or in a NUL-delimited
batch file, that would be great.

While special characters (such as newlines) in filenames are probably
bad practice, I believe that tools like scp(1) and sftp(1) should
be able to deal with them. At least the SFTP protocol is perfectly
capable of handling newlines in filenames, as I can delete a file
with a newline in its name using Nautilus over SFTP.

In case it matters, the client is a Fedora 30 Linux system, and the
server is running the latest OpenBSD snapshot.

Sincerely,

Demi

Attached Files:

Re: scp, sftp, and special characters in filenames [ In reply to ]
openssh at davidnewall
Nov 3, 2019, 4:43 PM
Post #2 of 9 (1399 views)
Permalink
On 4/11/19 3:29 am, Demi M. Obenour wrote:
> I have not been able to get scp(1) to download a file with a newline
> in its name. I know that scp(1) requires that remote filenames be
> escaped for the shell, but that leads to protocol errors.

I see something much worse:

$ sudo sh
# echo "#!/bin/sh
touch /tmp/b-ran" > /tmp/b
# chmod a+x /tmp/b
# exit
$ touch a 'a
b'
$ rm -f abc /tmp/b-ran
$ scp localhost:'a
b' abc
a 100% 0 0.0KB/s 00:00
$ ls /tmp/b-ran
/tmp/b-ran

Scp can be exploited to run commands, which I think is a problem.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: scp, sftp, and special characters in filenames [ In reply to ]
djm at mindrot
Nov 3, 2019, 6:07 PM
Post #3 of 9 (1399 views)
Permalink
On Mon, 4 Nov 2019, David Newall wrote:

> On 4/11/19 3:29 am, Demi M. Obenour wrote:
> > I have not been able to get scp(1) to download a file with a newline
> > in its name. I know that scp(1) requires that remote filenames be
> > escaped for the shell, but that leads to protocol errors.
>
> I see something much worse:
>
> $ sudo sh
> # echo "#!/bin/sh
> touch /tmp/b-ran" > /tmp/b
> # chmod a+x /tmp/b
> # exit
> $ touch a 'a
> b'
> $ rm -f abc /tmp/b-ran
> $ scp localhost:'a
> b' abc
> a 100% 0 0.0KB/s
> 00:00
> $ ls /tmp/b-ran
> /tmp/b-ran
>
> Scp can be exploited to run commands, which I think is a problem.

scp isn't being exploited, you're just running commands in your own
remote shell and all interpretation happens there - scp never sees
those characters.

This "feature" dates back to rcp from 1981 (scp is rcp in a nicer pair
of shoes) and isn't fixable without breaking scp's "protocol".

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: scp, sftp, and special characters in filenames [ In reply to ]
openssh at davidnewall
Nov 3, 2019, 6:54 PM
Post #4 of 9 (1399 views)
Permalink
On 4/11/19 12:37 pm, Damien Miller wrote:
> scp isn't being exploited, you're just running commands in your own
> remote shell and all interpretation happens there - scp never sees
> those characters.

I already understood what was going on.  The issue is, scp promises to
"cop[y] files between hosts on a network [using] ssh for data transfer."
It does not promise to also execute arbitrary commands.  When a file
copy program is used to execute arbitrary commands, that seems to almost
be the definition of an exploit.

The poor quality of command parsing (I mean, clumsy and awkward need to
quote and escape characters, even though the shell has already given scp
individual arguments) suggests why scp can be exploited in that manner. 
Although scp is speaking to sshd, obviously somewhere along the line one
of the components is translating that into a shell command line and thus
losing the identity of arguments.  That's just awful, and I should have
thought it was not at all necessary.  Am I missing something?

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: scp, sftp, and special characters in filenames [ In reply to ]
dtucker at dtucker
Nov 3, 2019, 7:28 PM
Post #5 of 9 (1399 views)
Permalink
On Mon, 4 Nov 2019 at 14:07, David Newall <openssh@davidnewall.com> wrote:

> [about scp] That's just awful, and I should have
> thought it was not at all necessary. Am I missing something?
>

If you're saying that the scp protocol is an unfixable mess then the
openssh team has been agreeing[0] with you for at least a decade and a
half. We fix what we can, but some parts can't be fixed.

[0] eg https://marc.info/?l=openssh-unix-dev&m=104157774216425&w=2

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: scp, sftp, and special characters in filenames [ In reply to ]
stu at spacehopper
Nov 4, 2019, 4:25 AM
Post #6 of 9 (1394 views)
Permalink
On 2019/11/04 14:28, Darren Tucker wrote:
> On Mon, 4 Nov 2019 at 14:07, David Newall <openssh@davidnewall.com> wrote:
>
> > [about scp] That's just awful, and I should have
> > thought it was not at all necessary. Am I missing something?
> >
>
> If you're saying that the scp protocol is an unfixable mess then the
> openssh team has been agreeing[0] with you for at least a decade and a
> half. We fix what we can, but some parts can't be fixed.
>
> [0] eg https://marc.info/?l=openssh-unix-dev&m=104157774216425&w=2

At various times I've tried to use sftp instead of scp but it never lasted
long. Remembering to type sftp instead of scp isn't so bad, but file
uploads are really hard to work with, I don't think it's possible to get
good finger memory for "echo 'put somefile' | sftp hostname:/tmp/" (and
it's even worse when copying out multiple files).

I'm sure it's already been considered, but from a user perspective it
would be very convenient if we could have essentially the command-line
interface of scp(1) (with divergence for filename escaping) but using
the SFTP protocol.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: scp, sftp, and special characters in filenames [ In reply to ]
nkadel at gmail
Nov 4, 2019, 5:51 PM
Post #7 of 9 (1394 views)
Permalink
On Mon, Nov 4, 2019 at 7:34 AM Stuart Henderson <stu@spacehopper.org> wrote:
>
> On 2019/11/04 14:28, Darren Tucker wrote:
> > On Mon, 4 Nov 2019 at 14:07, David Newall <openssh@davidnewall.com> wrote:
> >
> > > [about scp] That's just awful, and I should have
> > > thought it was not at all necessary. Am I missing something?
> > >
> >
> > If you're saying that the scp protocol is an unfixable mess then the
> > openssh team has been agreeing[0] with you for at least a decade and a
> > half. We fix what we can, but some parts can't be fixed.
> >
> > [0] eg https://marc.info/?l=openssh-unix-dev&m=104157774216425&w=2
>
> At various times I've tried to use sftp instead of scp but it never lasted
> long. Remembering to type sftp instead of scp isn't so bad, but file
> uploads are really hard to work with, I don't think it's possible to get
> good finger memory for "echo 'put somefile' | sftp hostname:/tmp/" (and
> it's even worse when copying out multiple files).
>
> I'm sure it's already been considered, but from a user perspective it
> would be very convenient if we could have essentially the command-line
> interface of scp(1) (with divergence for filename escaping) but using
> the SFTP protocol.

Filesystem limitations for files with ':' in them are a real problem.
The much simpler solution is that 'if it hurts when you put a pointy
thing in your eyeball, stop putting a pointy thin in your eyeball",
not to try to armor the eyeball. In this case, stop creating files
with ':' in the filename.

That said, David, you *might* have some success with using rsync over
SSH to transmit such files. But I make no promises.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: scp, sftp, and special characters in filenames [ In reply to ]
maniac.nl at gmail
Nov 5, 2019, 1:50 AM
Post #8 of 9 (1394 views)
Permalink
That's why rsync is the better alternative to scp/sftp ... it does uploads
and downloads with the same sane syntax, and does recursive, permissions,
and doesn't need to upload/download everything again when re-trying :)

rscynv -vPar <source> <dest>


On Mon, Nov 4, 2019 at 1:35 PM Stuart Henderson <stu@spacehopper.org> wrote:

> On 2019/11/04 14:28, Darren Tucker wrote:
> > On Mon, 4 Nov 2019 at 14:07, David Newall <openssh@davidnewall.com>
> wrote:
> >
> > > [about scp] That's just awful, and I should have
> > > thought it was not at all necessary. Am I missing something?
> > >
> >
> > If you're saying that the scp protocol is an unfixable mess then the
> > openssh team has been agreeing[0] with you for at least a decade and a
> > half. We fix what we can, but some parts can't be fixed.
> >
> > [0] eg https://marc.info/?l=openssh-unix-dev&m=104157774216425&w=2
>
> At various times I've tried to use sftp instead of scp but it never lasted
> long. Remembering to type sftp instead of scp isn't so bad, but file
> uploads are really hard to work with, I don't think it's possible to get
> good finger memory for "echo 'put somefile' | sftp hostname:/tmp/" (and
> it's even worse when copying out multiple files).
>
> I'm sure it's already been considered, but from a user perspective it
> would be very convenient if we could have essentially the command-line
> interface of scp(1) (with divergence for filename escaping) but using
> the SFTP protocol.
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>


--
Mark Janssen -- mark@sig-io.nl
Unix / Linux Open-Source and Internet Consultant
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Re: scp, sftp, and special characters in filenames [ In reply to ]
Jochen.Bern at binect
Nov 6, 2019, 1:21 AM
Post #9 of 9 (1393 views)
Permalink
On 11/04/2019 01:25 PM, Stuart Henderson wrote:
> I'm sure it's already been considered, but from a user perspective it
> would be very convenient if we could have essentially the command-line
> interface of scp(1) (with divergence for filename escaping) but using
> the SFTP protocol.

Agreed, but it essentially means to fix a server-side threat with
changes (that mainly happen) on the client side. In instances of
cross-organization data transfers with established multi-tenant servers,
that might not be a practical option.

On the other hand, if it were possible to set a server account's login
shell or ForceCommand to an "scpd", and thus completely cut out the
normal shell ...

(Yes, I'm aware that that'd likely entail re(?)implementing the entire
server-side wildcard expansion within that executable. Given that
different-OS shells' wildcard expansion is a known source of confusion,
that doesn't seem like the worst idea ever, though, if I may say so.)

Regards,
--
Jochen Bern
Systemingenieur

Binect GmbH
Robert-Koch-Straße 9
64331 Weiterstadt
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal