Hello,
This is probably a long email, but please bear with me. I plan to
submit a patch and would like to explain what I will do before doing
it so I don't lose time if there's some flaw in my plan.
I currently use sshfs to mount directories from some computers and a
NAS into other computers. I recently noticed that when copying some
files from one computer into one of these sshfs mounted directories
(supposedly preserving times) the files are losing the subsecond part
of mtime (and atime). So, for example, `stat foo` shows this locally:
Access: 2023-05-09 13:47:59.422975530 +0200
Modify: 2023-05-09 08:07:12.267263456 +0200
But `stat /mnt/whatever/foo` (the same file copied to the sshfs
mounted directory) shows:
Access: 2023-05-09 13:47:59.000000000 +0200
Modify: 2023-05-09 08:07:12.000000000 +0200
This is giving trouble to use those files indistinctly from both paths
by software that stores the mtime in a database to check if the files
changed since the last time they were processed.
Since sshfs basically uses sftp internally, I checked the openssh
implementation and indeed, it doesn't support subsecond times in file
attributes, so I'm considering adding support for that, but I'd like
to do it in a way that it can be approved officially, thus I'd like to
ask your opinions/ideas on the planned implementation.
As I said, I've been looking at the code and it seems there's no way
to do this "naturally" (adding u32 variables to the protocol stream
for the nsec parts of atime/mtime) since that would break the
compatibility with sftp clients/servers that don't expect those
variables in the stream.
I had a look at the protocol as defined in
https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13 and
that seems to have many differences (in some cases, even incompatible)
with the openssh sftp implementation, so I'll take the current code as
the main reference and just use
https://www.openssh.com/txt/draft-ietf-secsh-filexfer-02.txt as
documentation.
After a careful look at the protocol, it seems it could be done by making use of
SSH_FILEXFER_ATTR_EXTENDED attribute. Adding a pair or keys
("atime-ns" and "mtime-ns" for example) and since the value string
seems to allow holding binary data, it could contain the u32 values
directly in order to keep overhead as low as possible . Or even better
both values could be transmitted in a single "nsec-times" key/value
pair to keep overhead really to a minimum and just add both u32
variables to a separate sshbuf temporary variable and write that
sshbuf as the value string with sshbuf_put_stringb().
I think subsecond time support could be implemented in all file
attribs related protocol functions with the main changes involving the
stat_to_attrib(), attrib_to_stat(), decode_attrib(), encode_attrib(),
attrib_to_tv() and attrib_to_ts() functions mainly.
This way, sftp servers/clients that don't understand this "nsec-times"
extended attribute would just ignore it and keep working as up to now,
while those who understand them could use them.
Also, another option would be to make the server announce an
"nsec-attribs@openssh" extension so that if a client that can fill
nsec attributes is connecting to a server that doesn't announce
support for that, then we can keep the stream simple and not introduce
extended attributes unnecessarily . I see in `4. Protocol
Initialization` in the IETF draft it's supposed to be possible for the
client to send <extension data> just after the version in the FXP_INIT
packet, but looking at the sources it seems right now openssh's sftp
implementation doesn't send any extension from the client nor the
server tries to process it in any way. So this processing of client
supported extensions would have to be added in order to minimize the
overhead from the server if the client doesn't support subsecond times
(or just always submit them and let the extended attribute get
ignored).
What do you think of this plan? Does it sound like something that
could be accepted? Of course, if anyone can think of a better way to
implement this, I'd also be glad to read it.
And just for completion, if this gets a "go ahead", I'll also work on
making sshfs support this.
--
Antonio Larrosa
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
This is probably a long email, but please bear with me. I plan to
submit a patch and would like to explain what I will do before doing
it so I don't lose time if there's some flaw in my plan.
I currently use sshfs to mount directories from some computers and a
NAS into other computers. I recently noticed that when copying some
files from one computer into one of these sshfs mounted directories
(supposedly preserving times) the files are losing the subsecond part
of mtime (and atime). So, for example, `stat foo` shows this locally:
Access: 2023-05-09 13:47:59.422975530 +0200
Modify: 2023-05-09 08:07:12.267263456 +0200
But `stat /mnt/whatever/foo` (the same file copied to the sshfs
mounted directory) shows:
Access: 2023-05-09 13:47:59.000000000 +0200
Modify: 2023-05-09 08:07:12.000000000 +0200
This is giving trouble to use those files indistinctly from both paths
by software that stores the mtime in a database to check if the files
changed since the last time they were processed.
Since sshfs basically uses sftp internally, I checked the openssh
implementation and indeed, it doesn't support subsecond times in file
attributes, so I'm considering adding support for that, but I'd like
to do it in a way that it can be approved officially, thus I'd like to
ask your opinions/ideas on the planned implementation.
As I said, I've been looking at the code and it seems there's no way
to do this "naturally" (adding u32 variables to the protocol stream
for the nsec parts of atime/mtime) since that would break the
compatibility with sftp clients/servers that don't expect those
variables in the stream.
I had a look at the protocol as defined in
https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-13 and
that seems to have many differences (in some cases, even incompatible)
with the openssh sftp implementation, so I'll take the current code as
the main reference and just use
https://www.openssh.com/txt/draft-ietf-secsh-filexfer-02.txt as
documentation.
After a careful look at the protocol, it seems it could be done by making use of
SSH_FILEXFER_ATTR_EXTENDED attribute. Adding a pair or keys
("atime-ns" and "mtime-ns" for example) and since the value string
seems to allow holding binary data, it could contain the u32 values
directly in order to keep overhead as low as possible . Or even better
both values could be transmitted in a single "nsec-times" key/value
pair to keep overhead really to a minimum and just add both u32
variables to a separate sshbuf temporary variable and write that
sshbuf as the value string with sshbuf_put_stringb().
I think subsecond time support could be implemented in all file
attribs related protocol functions with the main changes involving the
stat_to_attrib(), attrib_to_stat(), decode_attrib(), encode_attrib(),
attrib_to_tv() and attrib_to_ts() functions mainly.
This way, sftp servers/clients that don't understand this "nsec-times"
extended attribute would just ignore it and keep working as up to now,
while those who understand them could use them.
Also, another option would be to make the server announce an
"nsec-attribs@openssh" extension so that if a client that can fill
nsec attributes is connecting to a server that doesn't announce
support for that, then we can keep the stream simple and not introduce
extended attributes unnecessarily . I see in `4. Protocol
Initialization` in the IETF draft it's supposed to be possible for the
client to send <extension data> just after the version in the FXP_INIT
packet, but looking at the sources it seems right now openssh's sftp
implementation doesn't send any extension from the client nor the
server tries to process it in any way. So this processing of client
supported extensions would have to be added in order to minimize the
overhead from the server if the client doesn't support subsecond times
(or just always submit them and let the extended attribute get
ignored).
What do you think of this plan? Does it sound like something that
could be accepted? Of course, if anyone can think of a better way to
implement this, I'd also be glad to read it.
And just for completion, if this gets a "go ahead", I'll also work on
making sshfs support this.
--
Antonio Larrosa
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev@mindrot.org
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
