Mailing List Archive

[Bug 801] Improve usability of scp by adding a "make path" option
http://bugzilla.mindrot.org/show_bug.cgi?id=801

Summary: Improve usability of scp by adding a "make path" option
Product: Portable OpenSSH
Version: -current
Platform: All
OS/Version: All
Status: NEW
Severity: enhancement
Priority: P3
Component: scp
AssignedTo: openssh-bugs@mindrot.org
ReportedBy: sdturne@us.ibm.com


I would like to see an option added to scp that would cause the creation of any
non-existent directories leading up to the target file. For example if I run the
following command:

scp -p file.dat remote:some/new/dir/file.dat

and any part of the "some/new/dir/" path does not exist, an error is returned.
It would be very useful to have a new option, say "-m", that would cause scp to
automatically create the missing directories without requiring a separate "ssh
remote mkdir -p some/new/dir/" invocation.

I see this as a performance enhancement as well as a convenience. I have
scenarios where I might have thousands of separate scp invocations, and it's
very expensive to run a separate invocation of "ssh remote mkdir -p" to insure
the target path exists (not to mention the fact that the "mkdir -p" command is
not portable to all the platforms I deal with).

I understand that scp is the follow-on to rcp, and that rcp has no such option.
It has been suggested that this might be a "reason" NOT to implement such a
feature in scp. How disappointing to think that a replacement tool can never
improve on the original! The GNU folks have never taken this approach, and as
long as the new feature doesn't break backward compatibility I can't see the
logic in this argument.

I might also note that scp already has logic to create directories (used within
the -r logic), so perhaps the problem is already partially solved.

I hope this seems like a reasonable request, and thanks for your consideration.
Steve Turner



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801

mouring@eviladmin.org changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |WONTFIX



------- Additional Comments From mouring@eviladmin.org 2004-02-27 02:47 -------
-p Preserves modification times, access times, and modes from the
original file.

Unable to use -p. And scp is not a replacement for rcp. It is a migration
path. Many places can move from rsh to ssh by just replacing one command. The
replacement for rcp is really 'sftp'.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From jmknoble@pobox.com 2004-02-27 03:18 -------
> I would like to see an option added to scp that would cause
> the creation of any non-existent directories leading up to the
> target file

Such functionality already exists, using ssh:

cat file.dat |ssh remote \
'mkdir -p some/new/dir && cd some/new/dir && cat >file.dat'

This also works in reverse:

(cd && mkdir -p some/new/dir && cd some/new/dir && \
ssh remote 'cat file.dat' >file.dat)

This functionality can easily be automated via a script at either
end of the conversation.

Alternatively, if all you want to do is replicate directory
structures, use 'tar':

tar cf - path/to/file.dat |ssh remote 'tar xvpf -'

or 'pax':

pax -w path/to/file.dat |ssh remote 'pax -rvp e'

Good luck.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801

sdturne@us.ibm.com changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|WONTFIX |



------- Additional Comments From sdturne@us.ibm.com 2004-02-27 03:53 -------
Ben, it does not appear that you understood my request.

I never suggested that "-p" be used to implement the "make path" feature;
obviously that option is already used. My suggestion was to use "-m".

Your statement that "scp is not a replacement for rcp" is simply not correct;
the documentation even states that it is.

How could I use sftp instead of scp? It doesn't operate anything close to what
I need, and it doesn't preserve attributes as does "scp -p".


Jim,

Your first two suggestions don't preserve the permissions or timestamps of the
source files; this is critical.

Your follow-on suggestions assume that my source file is already in the
directory hierachy that I need on the target; this is not the case.

There are most certainly umpteen other ways to "work around" my problem, but
aren't they all ugly, slow and kludgy compared to a simple:

scp -pm file.dat remote:some/new/dir/file.dat

? As I mentioned in a parallel discussion in comp.security.ssh, I've seen the
scp.c source code, and I can't envision this change requiring more than about 25
or 50 lines of code...



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From smoogen@lanl.gov 2004-02-27 05:49 -------
It may only take 30 lines, but scp is 'deprecated' by OpenBSD and the various
IETF proposed standards I have seen. That means that no new options/additions
will be added to scp and all work should be on the recommended replacement, sftp.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From sdturne@us.ibm.com 2004-02-27 07:22 -------
I have seen *zero* evidence of that, nor do I see any after searching the
OpenBSD website, www.ietf.org, or an exhaustive search of google. Can you
supply any evidence that scp is in fact considered deprecated by OpenBSD (or
anyone else)?

I hope you'll pardon me for saying so, but attempting to make scp "deprecated"
seems more than a bit ludicrous to me. As wonderful as I'm sure sftp is (I
haven't used it to any great extent), it is *not* a replacement for scp.

Between this bug report and the comp.security.ssh discussion, the amount of
negative feedback I've gotten for such a small request really surprises me.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801

djm@mindrot.org changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution| |WONTFIX



------- Additional Comments From djm@mindrot.org 2004-02-27 07:47 -------
You aren't listening: we aren't going to change scp.

Patches to teach sftp recursive operations and the few remaining bits of scp
commandline compatibility are welcome.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From dtucker@zip.com.au 2004-02-27 07:53 -------
The point is scp is a non-standard. The closest thing it has to a
specification is "what rcp does". The same command is used on both ends
of the connection, and it has to interoperate with other implementations.

New features are more likely in sftp, since the protocol is standardized
(well, a draft standard), and the client and server are decoupled. In
fact, you can already do what you want with the batch option, albeit in
a somewhat verbose way:

(echo -mkdir /tmp/dir; echo put -P file /tmp/dir/file) | sftp -b - localhost




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From markus@openbsd.org 2004-02-27 08:33 -------
there's no way to fix scp without breaking interoperbility.

better teach scp to speak sftp.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From sdturne@us.ibm.com 2004-02-27 09:41 -------
Damien Miller wrote:
>You aren't listening: we aren't going to change scp.

Really? I'm the one who's not listening? That's funny I kinda got the idea
that most of you weren't *reading*. I'm listening just fine, I'm just not
willing to let the issue drop without a decent explanation. Yours wasn't it.
Both you and Ben closed my request without sufficient explanation and without
the slightest shred of courtesy. You really should try working directly with
paying customers some time; you'd both do real well.

<deep breath>

Darren,

Your explanations are the first I've seen all day that hit the nail on the head,
and now I understand what the real danger is in changing scp. I'm now quite
happy to accept the fact that this issue won't be addressed, and thank you so
much for clearing it up. I applaud you sir.

I also want to apologize if some of my responses seem short-fused, but this
topic has been getting some snide remarks over in comp.security.ssh as well, and
I've been carrying some of that frustration over to this discussion. Sorry
'bout that.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From mouring@eviladmin.org 2004-02-27 10:01 -------
You must understand this type of thing comes up every 4 - 5 months, and after a
while it gets tiresome when people don't alway think through the changes they
are trying to push.

Examples from Bugzilla:

http://bugzilla.mindrot.org/show_bug.cgi?id=424
http://bugzilla.mindrot.org/show_bug.cgi?id=485
http://bugzilla.mindrot.org/show_bug.cgi?id=619
http://bugzilla.mindrot.org/show_bug.cgi?id=431

I'm sure if I dug through the OpenSSH-Unix-Dev@ list I could pull more examples.

<shrug> Others have already explained this. I'm going back to my day job.=)

- Ben




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From dtucker@zip.com.au 2004-02-27 10:19 -------
Created an attachment (id=553)
--> (http://bugzilla.mindrot.org/attachment.cgi?id=553&action=view)
Add scp to FAQ

Since this seems to be a FAQ :-)



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 801] Improve usability of scp by adding a "make path" option [ In reply to ]
http://bugzilla.mindrot.org/show_bug.cgi?id=801

djm@mindrot.org changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #553| |ok
Status| |



------- Additional Comments From djm@mindrot.org 2004-02-27 10:41 -------
(From update of attachment 553)
ok




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.