Mailing List Archive

Openssh server detect file transfer finished
Hi,

I've a directory that is watched for the event write-close.
When I get this event I assume that the file is received totally.

Is it correct? What is the behavior of openssh when a transfer failed? (
Server side ).

Micka,
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
A write-close can happen on transfer failure. It also can happen multiple times if you are doing a parallel transfer.

Thanks,
Kevin

________________________________________
From: openssh-unix-dev <openssh-unix-dev-bounces+kevin.fox=pnnl.gov@mindrot.org> on behalf of Micka <mickamusset@gmail.com>
Sent: Tuesday, December 1, 2020 4:21 PM
To: openssh-unix-dev@mindrot.org
Subject: Openssh server detect file transfer finished

Check twice before you click! This email originated from outside PNNL.


Hi,

I've a directory that is watched for the event write-close.
When I get this event I assume that the file is received totally.

Is it correct? What is the behavior of openssh when a transfer failed? (
Server side ).

Micka,
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mindrot.org%2Fmailman%2Flistinfo%2Fopenssh-unix-dev&amp;data=04%7C01%7CKevin.Fox%40pnnl.gov%7C6b854f0256c9479e8cb208d89658cb50%7Cd6faa5f90ae240338c0130048a38deeb%7C0%7C0%7C637424655407773492%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=lE%2BPTzUvvEIEVbL%2F93%2FS5%2FnjMEl%2F8KZuPnEbwNV%2F5Qg%3D&amp;reserved=0
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
Thanks,

Any idea how to detect a transfer failure?

Micka,

Le mer. 2 déc. 2020 à 01:27, Fox, Kevin M <Kevin.Fox@pnnl.gov> a écrit :

> A write-close can happen on transfer failure. It also can happen multiple
> times if you are doing a parallel transfer.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: openssh-unix-dev <openssh-unix-dev-bounces+kevin.fox=
> pnnl.gov@mindrot.org> on behalf of Micka <mickamusset@gmail.com>
> Sent: Tuesday, December 1, 2020 4:21 PM
> To: openssh-unix-dev@mindrot.org
> Subject: Openssh server detect file transfer finished
>
> Check twice before you click! This email originated from outside PNNL.
>
>
> Hi,
>
> I've a directory that is watched for the event write-close.
> When I get this event I assume that the file is received totally.
>
> Is it correct? What is the behavior of openssh when a transfer failed? (
> Server side ).
>
> Micka,
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
>
> https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mindrot.org%2Fmailman%2Flistinfo%2Fopenssh-unix-dev&amp;data=04%7C01%7CKevin.Fox%40pnnl.gov%7C6b854f0256c9479e8cb208d89658cb50%7Cd6faa5f90ae240338c0130048a38deeb%7C0%7C0%7C637424655407773492%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=lE%2BPTzUvvEIEVbL%2F93%2FS5%2FnjMEl%2F8KZuPnEbwNV%2F5Qg%3D&amp;reserved=0
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
On Wed, 2 Dec 2020, Micka wrote:

> Thanks,
>
> Any idea how to detect a transfer failure?

Have the client upload a zero-length "transfer succeeded" file after the
main file completes?

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
On Wed, 2 Dec 2020 at 13:36, Damien Miller <djm@mindrot.org> wrote:
> On Wed, 2 Dec 2020, Micka wrote:
> > Any idea how to detect a transfer failure?
>
> Have the client upload a zero-length "transfer succeeded" file after the
> main file completes?

Have the client upload into an unmonitored "in-progress" directory and
move (ie rename) the file into a monitored directory as the final
step?

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
On Wed, 2 Dec 2020, Darren Tucker wrote:

> On Wed, 2 Dec 2020 at 13:36, Damien Miller <djm@mindrot.org> wrote:
> > On Wed, 2 Dec 2020, Micka wrote:
> > > Any idea how to detect a transfer failure?
> >
> > Have the client upload a zero-length "transfer succeeded" file after the
> > main file completes?
>
> Have the client upload into an unmonitored "in-progress" directory and
> move (ie rename) the file into a monitored directory as the final
> step?

actually, a few people have asked about a client-side feature that
uploads files to a temporary name and then issues a rename once completed.

This would only help if using the OpenSSH sftp client though...

-d
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
On Wed, Dec 02, 2020 at 01:32:16PM +1100, Damien Miller wrote:
> On Wed, 2 Dec 2020, Micka wrote:
>
> > Thanks,
> >
> > Any idea how to detect a transfer failure?
>
> Have the client upload a zero-length "transfer succeeded" file after the
> main file completes?

When I did this in the mid 90s (for Oracle log shipping to a standby
database), I scp'd the datafile and then scp'd a checksum file and then
the receiving side would verify the checksum to ensure the file had
copied properly.

--

rgds
Stephen
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
On 02.12.20 01:21, Micka wrote:
> I've a directory that is watched for the event write-close.
> When I get this event I assume that the file is received totally.
> Is it correct?

No, an open FD *will* eventually be closed, by the OS when the process
terminates if need be. Unless the machine *crashes* in mid-operation, of
course ...

We do a lot of such transfers, and the (our) gold standard, if possible,
is to do an upload-then-rename under client control, with the further
processing stages on the server requiring a "non-temp name" match.

(Since you didn't specify how exactly you transfer the data: This can be
done with sftp and ssh(*), but not scp.)

(*) E.g., a la "uuencode tempfile < sourcefile | ssh user@host "uudecode
&& mv tempfile finalfile", not that I'd *recommend* such an approach ...

Regards
--
Jochen Bern
Systemingenieur

Binect GmbH
Re: Openssh server detect file transfer finished [ In reply to ]
Thanks, I'm using libcurl, and it's possible to rename after the
transfer is finished. I will test that.

Micka,

On Wed, Dec 2, 2020 at 10:50 AM Jochen Bern <Jochen.Bern@binect.de> wrote:
>
> On 02.12.20 01:21, Micka wrote:
> > I've a directory that is watched for the event write-close.
> > When I get this event I assume that the file is received totally.
> > Is it correct?
>
> No, an open FD *will* eventually be closed, by the OS when the process
> terminates if need be. Unless the machine *crashes* in mid-operation, of
> course ...
>
> We do a lot of such transfers, and the (our) gold standard, if possible,
> is to do an upload-then-rename under client control, with the further
> processing stages on the server requiring a "non-temp name" match.
>
> (Since you didn't specify how exactly you transfer the data: This can be
> done with sftp and ssh(*), but not scp.)
>
> (*) E.g., a la "uuencode tempfile < sourcefile | ssh user@host "uudecode
> && mv tempfile finalfile", not that I'd *recommend* such an approach ...
>
> Regards
> --
> Jochen Bern
> Systemingenieur
>
> Binect GmbH
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev@mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Re: Openssh server detect file transfer finished [ In reply to ]
Jochen Bern wrote:
> (*) E.g., a la "uuencode tempfile < sourcefile | ssh user@host "uudecode
> && mv tempfile finalfile", not that I'd *recommend* such an approach ...

ssh is 8-bit clean so no need for uuencode. One could:

ssh user@host 'f=`mktemp`; cat > "$f" && mv "$f" finalfile || rm "$f"' < srcfile

..or use a tool on top of ssh, like rsync, which does this and more.


//Peter
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev