Mailing List Archive

Matthew Case wrote:
[...]
> I've looked high and low and haven't really come up with anything
> definitive. Someone somewhere had mentioned fiddling with MTU settings,
> but I'm not really sure what that will accomplish as I am unfamiliar
> with what MTU is and does. If this question has been answered
> previously, I apologize ahead of time. Thanks!

This does sound like the MTU problem to which you refer. See
http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

Matt,

If you are using ssh do you need to use scp as well? Or is just plain copy ok?

On 10 March 2010 03:04, Darren Tucker <dtucker@zip.com.au> wrote:
> Matthew Case wrote:
> [...]
>>
>> I've looked high and low and haven't really come up with anything
>> definitive. Someone somewhere had mentioned fiddling with MTU settings, but
>> I'm not really sure what that will accomplish as I am unfamiliar with what
>> MTU is and does. If this question has been answered previously, I apologize
>> ahead of time. Thanks!
>
> This does sound like the MTU problem to which you refer.  See
> http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.
>
> --
> Darren Tucker (dtucker at zip.com.au)
> GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
>    Good judgement comes with experience. Unfortunately, the experience
> usually comes from bad judgement.
>

First and foremost, thank you to everyone for your responses. I checked
the MTU on both sides and it's currently 1500 so I'm assuming it's not a
mismatch. My VPN is a pair of old Netscreen 5xp boxes, and I can't find
anything relating to MTU or packet size in the configuration, but I'm
still looking.

Secondly, to answer your question John, There is no persistent
connection between the servers. I could feasibly set up an NFS share
between the two but I have a sneaking suspicion that if the problem is
some sort of packet mangling by the VPN during file transfers, the
actual mechanism used to transfer the file will be irrelevant. However,
I will set this up and test it and report back my results, most likely
next Monday.

On 3/12/2010 3:41 AM, John Morrison wrote:
> Matt,
>
> If you are using ssh do you need to use scp as well? Or is just plain copy ok?
>
> On 10 March 2010 03:04, Darren Tucker<dtucker@zip.com.au> wrote:
>
>> Matthew Case wrote:
>> [...]
>>
>>> I've looked high and low and haven't really come up with anything
>>> definitive. Someone somewhere had mentioned fiddling with MTU settings, but
>>> I'm not really sure what that will accomplish as I am unfamiliar with what
>>> MTU is and does. If this question has been answered previously, I apologize
>>> ahead of time. Thanks!
>>>
>> This does sound like the MTU problem to which you refer. See
>> http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.
>>
>> --
>> Darren Tucker (dtucker at zip.com.au)
>> GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
>> Good judgement comes with experience. Unfortunately, the experience
>> usually comes from bad judgement.
>>
>>
>
>
>

--

Matthew Case
Specialized Business Software
Software Engineer
SCJP 5 Certified
Phone: 440-542-9145
Fax: 440-542-9143




This message and any files transmitted with it may contain information that is privileged, confidential, and exempt from disclosure under applicable law. They are intended solely for the use of the intended recipient. If you are not the intended recipient, distributing, copying, disclosing, or reliance on the contents of this communication is strictly prohibited. If this has reached you in error, kindly destroy this message and notify the sender immediately. Thank you for your assistance.

We attempt to sweep harmful content (e.g. viruses) from e-mail and attachments, however we cannot guarantee their safety and can accept no liability for any resulting damage. The recipient is responsible to verify the safety of this message and any attachments before accepting them.

As with all networks, you need to have a policy of either:

i) always performing fragmentation as required and clearing the DF (don't fragment) bit on packets; or

ii) always allowing Path-MTU discovery to work by allowing ICMP un-reachable (subtype fragmentation required) packets to flow freely from all points in your network and over the VPN.

Note that you need to do one (or both) of these policies consistently on both sides of your network. Also note that the second option will give you the best performance and inter-operability with the rest of the internet.

A technical document on why this is required can be found here:

<http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml>

How to put all this into practice:

<http://www.cisco.com/en/US/tech/tk870/tk877/tk880/technologies_tech_note09186a008011a218.shtml>


--Paul


Matthew Case wrote:
> First and foremost, thank you to everyone for your responses. I checked
> the MTU on both sides and it's currently 1500 so I'm assuming it's not a
> mismatch. My VPN is a pair of old Netscreen 5xp boxes, and I can't find
> anything relating to MTU or packet size in the configuration, but I'm
> still looking.
>
> Secondly, to answer your question John, There is no persistent
> connection between the servers. I could feasibly set up an NFS share
> between the two but I have a sneaking suspicion that if the problem is
> some sort of packet mangling by the VPN during file transfers, the
> actual mechanism used to transfer the file will be irrelevant. However,
> I will set this up and test it and report back my results, most likely
> next Monday.
>
> On 3/12/2010 3:41 AM, John Morrison wrote:
> > Matt,
> >
> > If you are using ssh do you need to use scp as well? Or is just plain
> copy ok?
> >
> > On 10 March 2010 03:04, Darren Tucker<dtucker@zip.com.au> wrote:
> >
> >> Matthew Case wrote:
> >> [...]
> >>
> >>> I've looked high and low and haven't really come up with anything
> >>> definitive. Someone somewhere had mentioned fiddling with MTU
> settings, but
> >>> I'm not really sure what that will accomplish as I am unfamiliar with
> what
> >>> MTU is and does. If this question has been answered previously, I
> apologize
> >>> ahead of time. Thanks!
> >>>
> >> This does sound like the MTU problem to which you refer. See
> >> http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.
> >>
> >> --
> >> Darren Tucker (dtucker at zip.com.au)
> >> GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
> >> Good judgement comes with experience. Unfortunately, the experience
> >> usually comes from bad judgement.
> >>
> >>
> >
> >
> >
>
> --
>
> Matthew Case
> Specialized Business Software
> Software Engineer
> SCJP 5 Certified
> Phone: 440-542-9145
> Fax: 440-542-9143

Matt,

To do full diagnosis, do captures at both ends:
tcpdump -w client.pcap -s0 host <IP> and port 22
## so that we don't capture unnecessary stuff, if you are at the client, the <IP> should be the server side

At server side,
tcpdump -w server.pcap -s0 host <IP> and port 22
## so that we don't capture unnecessary stuff, if you are at the server, the <IP> should be the client side

Upload the {client,server}.pcap somewhere and provide the links, that way someone might be able to take a look at it.

This is one thing you can give it a try easily:

Certain network devices I have seen have bugs dealing with TCP SACK.
CentOS 5.3 comes enabled with TCP SACK.
As root, try issuing "sysctl -w net.ipv4.tcp_sack=0", then scp to check if turning SACK off resolve your issues.

Robin



________________________________________
From: listbounce@securityfocus.com [listbounce@securityfocus.com] On Behalf Of Matthew Case [mattcase@specializedbusinesssoftware.com]
Sent: Tuesday, March 09, 2010 12:44 PM
To: secureshell@securityfocus.com
Subject: Question about SCP stalling over VPN

Greetings,

I am attempting to SSH a series of relatively small files (tens of
megabytes) over a VPN connection to a remote linux server. I am able to
SSH between the servers without a problem, but when I use SCP to copy
the files I start seeing the following message more and more until
finally the copy grinds to a halt and eventually times out:

debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072

I've looked high and low and haven't really come up with anything
definitive. Someone somewhere had mentioned fiddling with MTU settings,
but I'm not really sure what that will accomplish as I am unfamiliar
with what MTU is and does. If this question has been answered
previously, I apologize ahead of time. Thanks!

I am running CentOS 5.3 with OpenSSH 4.3p2-29 on the server I am copying
the files from, and CentOS 5.4 with OpenSSH 4.3p2.36 on the server I am
copying to.
--

Matthew Case
Specialized Business Software
Software Engineer
SCJP 5 Certified
Phone: 440-542-9145
Fax: 440-542-9143




This message and any files transmitted with it may contain information that is privileged, confidential, and exempt from disclosure under applicable law. They are intended solely for the use of the intended recipient. If you are not the intended recipient, distributing, copying, disclosing, or reliance on the contents of this communication is strictly prohibited. If this has reached you in error, kindly destroy this message and notify the sender immediately. Thank you for your assistance.

We attempt to sweep harmful content (e.g. viruses) from e-mail and attachments, however we cannot guarantee their safety and can accept no liability for any resulting damage. The recipient is responsible to verify the safety of this message and any attachments before accepting them.

I have run into a similar problem, although I don't get any "rcvd
adjust" debug messages -- instead my scp transfers stall -- at a pretty
predictable place too (not exactly, but to within a few megabytes). My
problem ONLY started after I upgraded my kernel (and maybe some other
things :s, from 2.6.19 to 2.6.33). My NFS transfers also started
"stalling", for many minutes (5-15min usually), although they are
retried, and eventually finish. Again, I don't think ANY of these
problems ever happened with my old kernel for a couple years.

During the stalls, netstat says the tcp connection is still
ESTABLISHED, and when I Ctrl-C the stalled scp transfer, the sshd
server logs a "Received disconnect" message. No other debug messages
are shown for me. I'll try to get a tcpdump.


On Fri, 12 Mar 2010 09:13:47 -0500, Matthew Case wrote:
> First and foremost, thank you to everyone for your responses. I
> checked the MTU on both sides and it's currently 1500 so I'm assuming
> it's not a mismatch. My VPN is a pair of old Netscreen 5xp boxes, and
> I can't find anything relating to MTU or packet size in the
> configuration, but I'm still looking.
>
> Secondly, to answer your question John, There is no persistent
> connection between the servers. I could feasibly set up an NFS share
> between the two but I have a sneaking suspicion that if the problem
> is some sort of packet mangling by the VPN during file transfers, the
> actual mechanism used to transfer the file will be irrelevant.
> However, I will set this up and test it and report back my results,
> most likely next Monday.
>
> On 3/12/2010 3:41 AM, John Morrison wrote:
> > Matt,
> >
> > If you are using ssh do you need to use scp as well? Or is just
> > plain copy ok?
> >
> > On 10 March 2010 03:04, Darren Tucker<dtucker@zip.com.au> wrote:
> >
> >> Matthew Case wrote:
> >> [...]
> >>
> >>> I've looked high and low and haven't really come up with anything
> >>> definitive. Someone somewhere had mentioned fiddling with MTU
> >>> settings, but I'm not really sure what that will accomplish as I
> >>> am unfamiliar with what MTU is and does. If this question has
> >>> been answered previously, I apologize ahead of time. Thanks!
> >>>
> >> This does sound like the MTU problem to which you refer. See
> >> http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.
> >>

Err, with "-vvv" I get those exact same debug messages:

"debug2: channel 0: rcvd adjust 131072"

Lots of them. When trying to scp a 1G file to my server it consistently
stalls around 10MB (give or take a few).

What does that debug message mean, and why is it stalling, and why did
it work with my earlier kernel?

To be continued.


On Mon, 22 Mar 2010 13:02:47 -0400, Dennis Nezic wrote:
> I have run into a similar problem, although I don't get any "rcvd
> adjust" debug messages -- instead my scp transfers stall -- at a
> pretty predictable place too (not exactly, but to within a few
> megabytes). My problem ONLY started after I upgraded my kernel (and
> maybe some other things :s, from 2.6.19 to 2.6.33). My NFS transfers
> also started "stalling", for many minutes (5-15min usually), although
> they are retried, and eventually finish. Again, I don't think ANY of
> these problems ever happened with my old kernel for a couple years.
>
> During the stalls, netstat says the tcp connection is still
> ESTABLISHED, and when I Ctrl-C the stalled scp transfer, the sshd
> server logs a "Received disconnect" message. No other debug messages
> are shown for me. I'll try to get a tcpdump.
>
>
> On Fri, 12 Mar 2010 09:13:47 -0500, Matthew Case wrote:
> > First and foremost, thank you to everyone for your responses. I
> > checked the MTU on both sides and it's currently 1500 so I'm
> > assuming it's not a mismatch. My VPN is a pair of old Netscreen 5xp
> > boxes, and I can't find anything relating to MTU or packet size in
> > the configuration, but I'm still looking.
> >
> > Secondly, to answer your question John, There is no persistent
> > connection between the servers. I could feasibly set up an NFS
> > share between the two but I have a sneaking suspicion that if the
> > problem is some sort of packet mangling by the VPN during file
> > transfers, the actual mechanism used to transfer the file will be
> > irrelevant. However, I will set this up and test it and report back
> > my results, most likely next Monday.
> >
> > On 3/12/2010 3:41 AM, John Morrison wrote:
> > > Matt,
> > >
> > > If you are using ssh do you need to use scp as well? Or is just
> > > plain copy ok?
> > >
> > > On 10 March 2010 03:04, Darren Tucker<dtucker@zip.com.au> wrote:
> > >
> > >> Matthew Case wrote:
> > >> [...]
> > >>
> > >>> I've looked high and low and haven't really come up with
> > >>> anything definitive. Someone somewhere had mentioned fiddling
> > >>> with MTU settings, but I'm not really sure what that will
> > >>> accomplish as I am unfamiliar with what MTU is and does. If
> > >>> this question has been answered previously, I apologize ahead
> > >>> of time. Thanks!
> > >> This does sound like the MTU problem to which you refer. See
> > >> http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.
> > >>

The problem only happens with my wireless connection. (madwifi-ng on
the server side, kernel's b43 on the laptop.) With my wired ethernet
connection, the transfers all work. Sigh. I guess there is nothing more
this mailing list can do for me? :P.

(When wired, I'm sure it doesn't matter, but the rcvd adjust
messages steady out at 114688.)


On Tue, 23 Mar 2010 21:23:49 -0400, Dennis Nezic wrote:
> Err, with "-vvv" I get those exact same debug messages:
>
> "debug2: channel 0: rcvd adjust 131072"
>
> Lots of them. When trying to scp a 1G file to my server it
> consistently stalls around 10MB (give or take a few).
>
> What does that debug message mean, and why is it stalling, and why did
> it work with my earlier kernel?
>
> To be continued.
>
>
> On Mon, 22 Mar 2010 13:02:47 -0400, Dennis Nezic wrote:
> > I have run into a similar problem, although I don't get any "rcvd
> > adjust" debug messages -- instead my scp transfers stall -- at a
> > pretty predictable place too (not exactly, but to within a few
> > megabytes). My problem ONLY started after I upgraded my kernel (and
> > maybe some other things :s, from 2.6.19 to 2.6.33). My NFS transfers
> > also started "stalling", for many minutes (5-15min usually),
> > although they are retried, and eventually finish. Again, I don't
> > think ANY of these problems ever happened with my old kernel for a
> > couple years.
> >
> > During the stalls, netstat says the tcp connection is still
> > ESTABLISHED, and when I Ctrl-C the stalled scp transfer, the sshd
> > server logs a "Received disconnect" message. No other debug messages
> > are shown for me. I'll try to get a tcpdump.
> >
> >
> > On Fri, 12 Mar 2010 09:13:47 -0500, Matthew Case wrote:
> > > First and foremost, thank you to everyone for your responses. I
> > > checked the MTU on both sides and it's currently 1500 so I'm
> > > assuming it's not a mismatch. My VPN is a pair of old Netscreen
> > > 5xp boxes, and I can't find anything relating to MTU or packet
> > > size in the configuration, but I'm still looking.
> > >
> > > Secondly, to answer your question John, There is no persistent
> > > connection between the servers. I could feasibly set up an NFS
> > > share between the two but I have a sneaking suspicion that if the
> > > problem is some sort of packet mangling by the VPN during file
> > > transfers, the actual mechanism used to transfer the file will be
> > > irrelevant. However, I will set this up and test it and report
> > > back my results, most likely next Monday.
> > >
> > > On 3/12/2010 3:41 AM, John Morrison wrote:
> > > > Matt,
> > > >
> > > > If you are using ssh do you need to use scp as well? Or is just
> > > > plain copy ok?
> > > >
> > > > On 10 March 2010 03:04, Darren Tucker<dtucker@zip.com.au>
> > > > wrote:
> > > >> Matthew Case wrote:
> > > >> [...]
> > > >>
> > > >>> I've looked high and low and haven't really come up with
> > > >>> anything definitive. Someone somewhere had mentioned fiddling
> > > >>> with MTU settings, but I'm not really sure what that will
> > > >>> accomplish as I am unfamiliar with what MTU is and does. If
> > > >>> this question has been answered previously, I apologize ahead
> > > >>> of time. Thanks!
> > > >> This does sound like the MTU problem to which you refer. See
> > > >> http://www.snailbook.com/faq/mtu-mismatch.auto.html for
> > > >> details.
> > > >>