For a couple of years people have been reporting intermittent problems
with rsync over ssh freezing during a large transfer (typically
several GB). I have now gotten to the bottom of these problems, and
have written a small test program which demonstrates a deadlock in
sshd when doing large bi-directional transfers. Luckily the problem is
easily solved by modifying sshd to use socketpair() instead of
pipe(). Just remove "#define USE_PIPES 1" from near the bottom of
includes.h does the trick.
The test program, along with instructions for use and an explanation
of some of the pipe() bugs in various OSes that make this change
necessary is available from ftp://samba.org/pub/tridge/misc/sshtest.c
In past releases of rsync I added various buffering tricks in rsync
which reduced the likelyhood of this problem occuring, but didn't
eliminate it completely. I have removed those tricks in rsync 2.4.0 so
you may find an increase in the number of bug reports regarding rsync
over ssh. I removed the tricks as they caused excessive memory usage
(to the point of memory starvation) in some cases and are completely
unnecessary once sshd is modified to use socketpair().
I think the combination of pipe/select bugs on various OSes means that
you can't work around this problem by playing with non-blocking pipes
or the size of writes (except by dropping the size by 1).
I'm also amazed at the number of pipe/select bugs in the various OSes
I tested - Linux 2.2 was particularly bad (fixed in 2.3). I really
expected pipes to be pretty well defined and bug-free by now!
Cheers, Tridge
PS: I am not on this list, I'm just using this address as noted in the
README.
with rsync over ssh freezing during a large transfer (typically
several GB). I have now gotten to the bottom of these problems, and
have written a small test program which demonstrates a deadlock in
sshd when doing large bi-directional transfers. Luckily the problem is
easily solved by modifying sshd to use socketpair() instead of
pipe(). Just remove "#define USE_PIPES 1" from near the bottom of
includes.h does the trick.
The test program, along with instructions for use and an explanation
of some of the pipe() bugs in various OSes that make this change
necessary is available from ftp://samba.org/pub/tridge/misc/sshtest.c
In past releases of rsync I added various buffering tricks in rsync
which reduced the likelyhood of this problem occuring, but didn't
eliminate it completely. I have removed those tricks in rsync 2.4.0 so
you may find an increase in the number of bug reports regarding rsync
over ssh. I removed the tricks as they caused excessive memory usage
(to the point of memory starvation) in some cases and are completely
unnecessary once sshd is modified to use socketpair().
I think the combination of pipe/select bugs on various OSes means that
you can't work around this problem by playing with non-blocking pipes
or the size of writes (except by dropping the size by 1).
I'm also amazed at the number of pipe/select bugs in the various OSes
I tested - Linux 2.2 was particularly bad (fixed in 2.3). I really
expected pipes to be pretty well defined and bug-free by now!
Cheers, Tridge
PS: I am not on this list, I'm just using this address as noted in the
README.