Mailing List Archive

rapier wrote:
> I'm open to any and all suggestions, modifications, criticism and so forth.

Do you think there are some ways to accomplish same or similar
functionality also for SFTP? Perhaps that could help acceptance
for some SCP development.


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 4/1/21 2:28 PM, Peter Stuge wrote:
> rapier wrote:
>> I'm open to any and all suggestions, modifications, criticism and so forth.
>
> Do you think there are some ways to accomplish same or similar
> functionality also for SFTP? Perhaps that could help acceptance
> for some SCP development.

SFTP already has a resume feature with reput and reget (aka put -a and
get -a). The main difference is that it doesn't check to see if file
chunks are identical before appending. I think that would be handy and I
might do that at some point.

Chris
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

If you’re looking for incremental copy capabilities, why not just use rsync, which already runs nicely over SSH?

On Apr 1, 2021, at 10:50 AM, rapier <rapier@psc.edu> wrote:
> Howdy all,
>
> I know development on SCP is discouraged but being that it's still in wide use I thought I would do some work some of my users have been asking for and allow SCP to resume from a partial transfer.
>
> When enabled the resume function will compare the source and target files. The source host will and target host will pass MD5 hashes back and forth in order to compare the hash of the file fragment on the target and the file up to the length of the source file. If these hashes match the source will seek to the appropriate byte position and start sending the rest of the file. This will then be appended to the target.
>
> Example:
> An scp transfer of a 128,000 byte file terminates after transferring 64,000 bytes. The user then issues the SCP command again with the -R (resume) option. The hash of the 64k fragment on the target is compared to the hash of the source file up to 64k. The hashes match so the source seeks to the 64,001th byte and then starts send the remainder of the file. This is written to a temp file on the target which then appends it to the target after the end of the transfer.
>
> If the hash doesn't match then then entire source file is copied to the target and replaces the fragment.
>
> In the event that the source and target files have the same size and same hash then the file is skipped (no transfer of data). This end sup being handy if the user is doing a recursive transfer or using a wildcard.
>
> Both sides have to be using a resume enabled SCP for this to work. Since scp uses the first scp in the users path I've added an option that lets users define which scp to use on the remote side. This can be set with '-s' e.g. 'scp -R -s /opt/openssh/bin/scp' foo* you@host:~/foofiles/'
>
> The code is available at https://github.com/rapier1/openssh-portable in the resume-scp branch. The patch file is ~1k lines long so I didn't think posting it here would be the right move.
>
> Note: This is the first functional implementation of this so there are a lot of debug statement and I am sure there are ways of improving this. I also know that users can just use rsync. That said, this was a deliverable and the users wanted this. So I did it. Anyway, I'm open to any and all suggestions, modifications, criticism and so forth.
>
> Thanks,
>
> Chris

--
Ron Frederick
ronf@timeheart.net



_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Rsync already has the behavior you speak of (transfer only blocks that differ). Any reason why `rsync -P` won't work for your case (with or without `--checksum`)?

--
jmk

> On Apr 1, 2021, at 14:49, rapier <rapier@psc.edu> wrote:
>
> ?
>
>> On 4/1/21 2:28 PM, Peter Stuge wrote:
>> rapier wrote:
>>> I'm open to any and all suggestions, modifications, criticism and so forth.
>> Do you think there are some ways to accomplish same or similar
>> functionality also for SFTP? Perhaps that could help acceptance
>> for some SCP development.
>
> SFTP already has a resume feature with reput and reget (aka put -a and get -a). The main difference is that it doesn't check to see if file chunks are identical before appending. I think that would be handy and I might do that at some point.
>
> Chris
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

User demand and it's a deliverable for a project. Like I said in my
first message I know that rsync can do basically the same thing but ya
know, deliverables need to be delivered. Just thought I'd share the
results.

On 4/1/21 3:00 PM, Ron Frederick wrote:
> If you’re looking for incremental copy capabilities, why not just use rsync, which already runs nicely over SSH?
>
> On Apr 1, 2021, at 10:50 AM, rapier <rapier@psc.edu> wrote:
>> Howdy all,
>>
>> I know development on SCP is discouraged but being that it's still in wide use I thought I would do some work some of my users have been asking for and allow SCP to resume from a partial transfer.
>>
>> When enabled the resume function will compare the source and target files. The source host will and target host will pass MD5 hashes back and forth in order to compare the hash of the file fragment on the target and the file up to the length of the source file. If these hashes match the source will seek to the appropriate byte position and start sending the rest of the file. This will then be appended to the target.
>>
>> Example:
>> An scp transfer of a 128,000 byte file terminates after transferring 64,000 bytes. The user then issues the SCP command again with the -R (resume) option. The hash of the 64k fragment on the target is compared to the hash of the source file up to 64k. The hashes match so the source seeks to the 64,001th byte and then starts send the remainder of the file. This is written to a temp file on the target which then appends it to the target after the end of the transfer.
>>
>> If the hash doesn't match then then entire source file is copied to the target and replaces the fragment.
>>
>> In the event that the source and target files have the same size and same hash then the file is skipped (no transfer of data). This end sup being handy if the user is doing a recursive transfer or using a wildcard.
>>
>> Both sides have to be using a resume enabled SCP for this to work. Since scp uses the first scp in the users path I've added an option that lets users define which scp to use on the remote side. This can be set with '-s' e.g. 'scp -R -s /opt/openssh/bin/scp' foo* you@host:~/foofiles/'
>>
>> The code is available at https://github.com/rapier1/openssh-portable in the resume-scp branch. The patch file is ~1k lines long so I didn't think posting it here would be the right move.
>>
>> Note: This is the first functional implementation of this so there are a lot of debug statement and I am sure there are ways of improving this. I also know that users can just use rsync. That said, this was a deliverable and the users wanted this. So I did it. Anyway, I'm open to any and all suggestions, modifications, criticism and so forth.
>>
>> Thanks,
>>
>> Chris
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

User demand and delivery requirements of the project. You know how that
goes.

On 4/1/21 3:04 PM, Jim Knoble wrote:
> Rsync already has the behavior you speak of (transfer only blocks that differ). Any reason why `rsync -P` won't work for your case (with or without `--checksum`)?
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

For anyone that did look at the code earlier today there was a bug with
how it was dealing with permission denied errors in resume mode. That's
been resolved in the latest push.

Chris
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

rapier wrote:
> User demand and it's a deliverable for a project. Like I said in my first
> message I know that rsync can do basically the same thing but ya know,
> deliverables need to be delivered. Just thought I'd share the results.

I would be inclined to act as a BOFH and replace scp with a script
wrapper that called rsync. "There! I deliverd an scp that handles
resume. Problem solved."

Bob
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Fri, 2 Apr 2021, Bob Proulx wrote:

> I would be inclined to act as a BOFH and replace scp with

I keep it… it’s useful to push to machines before they have rsync
(e.g. to push updated sources.list, keys, etc).

> a script wrapper that called rsync. "There! I deliverd an scp that
> handles resume. Problem solved."

Definitely! (Except… one point against rsync is its licence.)

I’ve named mine rcp though.

https://evolvis.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=shellsnippets/shellsnippets.git;a=blob;f=mksh/rcp;hb=HEAD

It does quite some things (e.g. -a --numeric-ids) by default
but parses options, to some degree, to disable them again if
needed. (Oh, I see I need to add the recent additions to usage…)

Enjoy,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

*************************************************

Mit dem tarent-Newsletter nichts mehr verpassen: www.tarent.de/newsletter

*************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Thorsten Glaser wrote:
> Bob Proulx wrote:
> > I would be inclined to act as a BOFH and replace scp with
>
> I keep it... it's useful to push to machines before they have rsync
> (e.g. to push updated sources.list, keys, etc).

Rename it (on Debian dpkg-divert it) to scp.real? That would be fine.

> > a script wrapper that called rsync. "There! I deliverd an scp that
> > handles resume. Problem solved."

> I've named mine rcp though.

And the circle is now complete. Though I guess there is no danger of
it name colliding with the original real rcp anymore.

The original problem was it needed to be called for it to be "scp" in
order to meet the job spec. I would push back against that.

> https://evolvis.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=shellsnippets/shellsnippets.git;a=blob;f=mksh/rcp;hb=HEAD
>
> It does quite some things (e.g. -a --numeric-ids) by default
> but parses options, to some degree, to disable them again if
> needed. (Oh, I see I need to add the recent additions to usage...)

Yes. Something like that. :-)

Bob
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Fri, 2 Apr 2021, Bob Proulx wrote:

> > I keep it... it's useful to push to machines before they have rsync
> > (e.g. to push updated sources.list, keys, etc).
>
> Rename it (on Debian dpkg-divert it) to scp.real? That would be fine.

That would be an option.

> > > a script wrapper that called rsync. "There! I deliverd an scp that
> > > handles resume. Problem solved."
>
> > I've named mine rcp though.
>
> And the circle is now complete.

That was the intention ;-)

> Though I guess there is no danger of
> it name colliding with the original real rcp anymore.

On some systems, rcp calls scp by default which can be irritating
when I don’t know whether I had placed mine there yet or not…
I learnt this later.

> The original problem was it needed to be called for it to be "scp" in
> order to meet the job spec.

Considering rsync’s vastly differing semantics especially regarding
trailing slashes for directories in source, destination and both,
I deliberately did not name it scp-alike. (I lost my eMail archives
once due to missing a slash, with rcp --delete. No big loss ofc, had
backups of anything really important.)

bye,
//mirabilos
--
Infrastrukturexperte • tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn • http://www.tarent.de/
Telephon +49 228 54881-393 • Fax: +49 228 54881-235
HRB AG Bonn 5168 • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

*************************************************

Mit dem tarent-Newsletter nichts mehr verpassen: www.tarent.de/newsletter

*************************************************
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Fri, Apr 2, 2021 at 11:52 PM Bob Proulx <bob@proulx.com> wrote:
>
> rapier wrote:
> > User demand and it's a deliverable for a project. Like I said in my first
> > message I know that rsync can do basically the same thing but ya know,
> > deliverables need to be delivered. Just thought I'd share the results.
>
> I would be inclined to act as a BOFH and replace scp with a script
> wrapper that called rsync. "There! I deliverd an scp that handles
> resume. Problem solved."
>
> Bob

Rewriting security sensitive binaries to solve a problem that is
already solved at the command line, with a great deal more flexibility
and power, is also usually a really, really bad idea.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 4/3/21 10:41 AM, Nico Kadel-Garcia wrote:
> On Fri, Apr 2, 2021 at 11:52 PM Bob Proulx <bob@proulx.com> wrote:
>>
>> rapier wrote:
>>> User demand and it's a deliverable for a project. Like I said in my first
>>> message I know that rsync can do basically the same thing but ya know,
>>> deliverables need to be delivered. Just thought I'd share the results.
>>
>> I would be inclined to act as a BOFH and replace scp with a script
>> wrapper that called rsync. "There! I deliverd an scp that handles
>> resume. Problem solved."
>>
>> Bob
>
> Rewriting security sensitive binaries to solve a problem that is
> already solved at the command line, with a great deal more flexibility
> and power, is also usually a really, really bad idea.

Been doing that for a long time as this is part of HPN-SSH. I have
people I work with at CERT and Trusted CI to help with the code audits.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 4/2/21 11:42 PM, Bob Proulx wrote:
> rapier wrote:
>> User demand and it's a deliverable for a project. Like I said in my first
>> message I know that rsync can do basically the same thing but ya know,
>> deliverables need to be delivered. Just thought I'd share the results.
>
> I would be inclined to act as a BOFH and replace scp with a script
> wrapper that called rsync. "There! I deliverd an scp that handles
> resume. Problem solved."
>
> Bob
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

While not a bad idea (and I did think about) this is part of an NSF
grant (I'm the PI - NSF Award #2004012). Since I want to keep getting
grants I need to do the work (and I'd be screwing taxpayers out of their
money). This is one of the smaller deliverables but I thought it would
be worth sharing with the community.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

> On 01 Apr 2021, at 19:50 , rapier <rapier@psc.edu> wrote:
>
> The code is available at https://github.com/rapier1/openssh-portable in the resume-scp branch. The patch file is ~1k lines long so I didn't think posting it here would be the right move.

Given the date this was released, I had to double check… I’m just surprised it didn’t include a “—-parallel=<number>” to speedup the transfers over latency links… much more needed for me transferring data continuously between the northern and southern hemispheres ;(
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

Parallelizing the transfer isn't a bad idea assuming the goal is to
saturate the path. Its not super hard to do if you are doing it a per
file basis but optimizing the queue can be hard to do right. That said,
using fpart as a feeder to the queues would work. It's how parsyncfp works.

Anyway, the HPN-SSH works hard to saturate the pipe already by using
optimized buffers, parallel ciphers (which need more work - currently
trying to get AES-NI parallelized for CTR and chacha20), disabling the
cipher (post authentication) for non-sensitive transfers, and so forth.
I can push data at around 4Gb/s with a single stream between
transcontinental hosts.

On 4/3/21 11:30 AM, hvjunk wrote:
>
>
>> On 01 Apr 2021, at 19:50 , rapier <rapier@psc.edu> wrote:
>>
>> The code is available at https://github.com/rapier1/openssh-portable in the resume-scp branch. The patch file is ~1k lines long so I didn't think posting it here would be the right move.
>
> Given the date this was released, I had to double check… I’m just surprised it didn’t include a “—-parallel=<number>” to speedup the transfers over latency links… much more needed for me transferring data continuously between the northern and southern hemispheres ;(
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 4/1/21 1:50 PM, rapier wrote:
> Howdy all,
>
> I know development on SCP is discouraged but being that it's still in wide use
> I thought I would do some work some of my users have been asking for and allow
> SCP to resume from a partial transfer.

Would it be possible to instead reimplement SCP in terms of SFTP, and then add
this feature to SFTP? My understanding is that such a re-implementation is
something many people have wanted for quite a while.

Of course, this might very well be out of scope for the project, which would
be fine.

> When enabled the resume function will compare the source and target files. The
> source host will and target host will pass MD5 hashes back and forth in order
> to compare the hash of the file fragment on the target and the file up to the
> length of the source file. If these hashes match the source will seek to the
> appropriate byte position and start sending the rest of the file. This will
> then be appended to the target.

I suggest using a better hash than MD5, which is considered broken. Blake2b is
both faster and much more secure.

Sincerely,

Demi

On 4/3/21 8:10 PM, Demi Marie Obenour wrote:
> On 4/1/21 1:50 PM, rapier wrote:
>> Howdy all,
>>
>> I know development on SCP is discouraged but being that it's still in wide use
>> I thought I would do some work some of my users have been asking for and allow
>> SCP to resume from a partial transfer.
>
> Would it be possible to instead reimplement SCP in terms of SFTP, and then add
> this feature to SFTP? My understanding is that such a re-implementation is
> something many people have wanted for quite a while.
>
> Of course, this might very well be out of scope for the project, which would
> be fine.

Honestly, after working on the SCP code I do support that idea. SCP
really depends on an in band control protocol that can get out of sync
and freeze the transfer process. The right thing might be to use SCP as
a wrapper for SFTP. Mostly to maintain user experience and existent
scripts. I may look at that depending on time and progress on other
aspects of this project.

> I suggest using a better hash than MD5, which is considered broken. Blake2b is
> both faster and much more secure.

I've been looking at several hashes for this: blake2, sha1, md5, and
xxhash. MD5 was the first pass at implementation and I've since changed
to using EVP contexts. I fully expect to go with blake2 in the end but I
need to run more performance tests. The hashing ends up being one of the
more expensive operations (especially on very large files (100s of MB to
GB)) so that section is still subject to change.

I am trying to figure out how to reduce the number of hash operations.
Let me lay it out to see if anyone has ideas (aside from using rsync -
which I fully support).

Source: stat file, get hash, send control sequence 'C' to target.
(Cfilemode filesize(s), hash(s), filename(s))
Target: Receive control sequence.
If target exists
compute hash(t)
If hash(t) == hash(s)
skip file (send skip control sequence 'S' to source)
If (hash(t) != hash(s)
send control sequence 'R' to source
(Rfilemode, filesize(t), hash(t))
Source: Receive control sequence from source
If control == 'S'
skip file
If control == 'R'
compute hash(r) of target file to filesize(t)
if hash(r) == hash(t)
file fragments match
mode = R (for resume)
bytes = filesize(s) - filesize(t)
If hash(r) != hash(t)
fragments do not match
mode = C (for create)
bytes = filesize(t)
send control to target (mode, bytes)
Target: Receive control seq from source
if mode == R
write bytes to temp file
append temp file to target
if mode == C
write bytes to file

I think rsync only computes hashes if the modification time, files
sizes, and other file stat data is different. I thought about doing that
but since you can rename the target with scp that won't work.

Anyway, if anyone has an ideas on reducing the steps, hashes. etc let me
know. I also cannot figure out why I can append directly to the target
file. After opening the file I'd seek to the end but the bytes would
still start at the 0th byte. I'm probably missing something in atomicio.
Writing the temp file and then appending works and it's not taking up a
lot of cycles but it doesn't feel like the 'right' way to do it.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

--
jmk

> On Apr 4, 2021, at 11:59, rapier <rapier@psc.edu> wrote:
>
> [...] I am trying to figure out how to reduce the number of hash operations. Let me lay it out to see if anyone has ideas (aside from using rsync - which I fully support).

Have you considered implementing this using librsync, which is LGPL-2.1?
https://github.com/librsync/librsync

This is the delta library used by rdiff-backup (which you might consider looking at for inspiration (it's Python, but is GPL v2, so use care)).

> [...] I think rsync only computes hashes if the modification time, files sizes, and other file stat data is different. I thought about doing that but since you can rename the target with scp that won't work.

(This behavior can be changed by using the `--checksum` option with rsync, which hashes a file on both ends to determine sameness).

> [...] I also cannot figure out why I can append directly to the target file. After opening the file I'd seek to the end but the bytes would still start at the 0th byte. I'm probably missing something in atomicio. Writing the temp file and then appending works and it's not taking up a lot of cycles but it doesn't feel like the 'right' way to do it.

This certainly does sound suboptimal. Maybe if you post the code?

Cheers.

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 4/4/21 9:45 PM, Jim Knoble wrote:
>> [...] I also cannot figure out why I can append directly to the target
>> file. After opening the file I'd seek to the end but the bytes would
>> still start at the 0th byte. I'm probably missing something in
>> atomicio. Writing the temp file and then appending works and it's not
>> taking up a lot of cycles but it doesn't feel like the 'right' way to
>> do it.
>
> This certainly does sound suboptimal. Maybe if you post the code?

In context the you can see the code at
https://github.com/rapier1/openssh-portable/blob/resume-scp/scp.c

The diff is available at: https://pastebin.com/icXjV2JP

Honestly, I have no idea why the direct append isn't working. In 'sink'
the code opens ofd with an open. I had initially tried doing an lseek to
the end of the file which didn't work. Then I tried open the file with
O_APPEND and that didn't work either. I think the issue is in atomicio6
but modifying that would be more intrusive than I would like for this
issue.

Chris
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Sat, 3 Apr 2021, Demi Marie Obenour wrote:

> On 4/1/21 1:50 PM, rapier wrote:
> > Howdy all,
> >
> > I know development on SCP is discouraged but being that it's still
> > in wide use I thought I would do some work some of my users have
> > been asking for and allow SCP to resume from a partial transfer.
>
> Would it be possible to instead reimplement SCP in terms of SFTP, and then add
> this feature to SFTP? My understanding is that such a re-implementation is
> something many people have wanted for quite a while.

Yes, and there are patches to do this awaiting review:

https://github.com/openssh/openssh-portable/pull/194

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 4/5/21 6:22 PM, Damien Miller wrote:
> On Sat, 3 Apr 2021, Demi Marie Obenour wrote:
>
>> On 4/1/21 1:50 PM, rapier wrote:
>>> Howdy all,
>>>
>>> I know development on SCP is discouraged but being that it's still
>>> in wide use I thought I would do some work some of my users have
>>> been asking for and allow SCP to resume from a partial transfer.
>>
>> Would it be possible to instead reimplement SCP in terms of SFTP, and then add
>> this feature to SFTP? My understanding is that such a re-implementation is
>> something many people have wanted for quite a while.
>
> Yes, and there are patches to do this awaiting review:
>
> https://github.com/openssh/openssh-portable/pull/194
>
> -d

Sooner those get merged, the better, IMO. I for one will celebrate the end
of the SCP protocol.

Sincerely,

Demi

On 4/5/21 6:22 PM, Damien Miller wrote:
> On Sat, 3 Apr 2021, Demi Marie Obenour wrote:
>
>> On 4/1/21 1:50 PM, rapier wrote:
>>> Howdy all,
>>>
>>> I know development on SCP is discouraged but being that it's still
>>> in wide use I thought I would do some work some of my users have
>>> been asking for and allow SCP to resume from a partial transfer.
>>
>> Would it be possible to instead reimplement SCP in terms of SFTP, and then add
>> this feature to SFTP? My understanding is that such a re-implementation is
>> something many people have wanted for quite a while.
>
> Yes, and there are patches to do this awaiting review:
>
> https://github.com/openssh/openssh-portable/pull/194

That would be awesome. The more I work with the SCP code the more I've
come to dislike it.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On 4/6/21 10:27 AM, rapier wrote:
>
>
> On 4/5/21 6:22 PM, Damien Miller wrote:
>> On Sat, 3 Apr 2021, Demi Marie Obenour wrote:
>>
>>> On 4/1/21 1:50 PM, rapier wrote:
>>>> Howdy all,
>>>>
>>>> I know development on SCP is discouraged but being that it's still
>>>> in wide use I thought I would do some work some of my users have
>>>> been asking for and allow SCP to resume from a partial transfer.
>>>
>>> Would it be possible to instead reimplement SCP in terms of SFTP, and
>>> then add
>>> this feature to SFTP?  My understanding is that such a
>>> re-implementation is
>>> something many people have wanted for quite a while.
>>
>> Yes, and there are patches to do this awaiting review:
>>
>> https://github.com/openssh/openssh-portable/pull/194
>
> That would be awesome. The more I work with the SCP code the more I've
> come to dislike it.

Looking at the performance - on my systems sftp seems to be a bit slower
than scp when dealing with a lot of small files. Not sure why this is
the case as I haven't looked at the sftp code in years.

For 100MB over 2666 files:
[rapier test] time sftp -r me@foo:~/testdir testdir
real 0m10.902s
user 0m1.532s
sys 0m2.679s

[rapier test] time scp -r me@foo:~/testdir testdir
real 0m5.915s
user 0m0.883s
sys 0m1.497s

A test with 6.8G over 19 files has nearly identical performance.

That said, this may be "good enough" for people or it could be an issue
with my somewhat older test hosts. Dunno but I wanted to mention it.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev