Mailing List Archive: Net::SFTP Slowness

Net::SFTP Slowness

rolfm at denison

Sep 28, 2007, 10:19 AM

Post #1 of 21 (15904 views)

One of the weaknesses of Bricolage seems to be the STFP connection.
For those who haven't used it, it's very slow on large media files.
We have a a 44MB video that takes a minute and a half to publish to
the production server with bric_queued. 1-2MB image files also take
10 seconds or more. Once you get 200 people publishing, well, that
causes a backlog.

We've installed Math::BigIng::GMP, and that is primarily for reducing
login time - we're fine on that. The actual file transfer is what
seems to be slow.

At any rate, we're looking right at Net::SFTP as the potential
culprit. Anyone have any thoughts or experiences with it not living
up to expectations? Is there something we might be doing wrong? The
Bricolage server and the production server are both hooked into the
gigabit backbone. Bandwidth is not a problem, and that's been
verified with other tests.

We're thinking about rewriting Bric::Util::Trans::SFTP.pm to link up
to Net::SFTP::Foreign, which uses the machine's ssh client that it
find in the path, and seems to get much better transfer times. The
main issue I can see with that is that Net::SFTP handles usernames and
passwords better than Net::SFTP::Foreign, the latter requiring Expect
code to set up the password exchange.

Any thoughts on this? Net::SFTP::Foreign seems to be pretty new, and
I doubt it would have been considered when SFTP functionality was added.

-Matt

Re: Net::SFTP Slowness [ In reply to ]

david at kineticode

Sep 28, 2007, 11:51 AM

Post #2 of 21 (15707 views)

On Sep 28, 2007, at 11:19, rolfm@denison.edu wrote:

> At any rate, we're looking right at Net::SFTP as the potential
> culprit. Anyone have any thoughts or experiences with it not
> living up to expectations? Is there something we might be doing
> wrong? The Bricolage server and the production server are both
> hooked into the gigabit backbone. Bandwidth is not a problem, and
> that's been verified with other tests.

Have you tried tweaking SFTP_MOVER_CIPHER in bricolage.conf?

Best,

David

Re: Net::SFTP Slowness [ In reply to ]

rolfm at denison

Sep 28, 2007, 12:01 PM

Post #3 of 21 (15704 views)

On Sep 28, 2007, at 3:51 PM, David E. Wheeler wrote:

> On Sep 28, 2007, at 11:19, rolfm@denison.edu wrote:
> <rant>
> Have you tried tweaking SFTP_MOVER_CIPHER in bricolage.conf?

No. Might make a difference?

-Matt

RE: Net::SFTP Slowness [ In reply to ]

D-Beaudet at NGA

Sep 28, 2007, 12:09 PM

Post #4 of 21 (15706 views)

Going from the slowest to the fastest cipher could possibly double or
triple your speed, but from what it sounded like, you have other issues
going on if you're only getting 100 KB / second transfer rates. That's
a couple orders of magnitude slower than command line SCP over gigabit
which should be on the order of 20 MB / second.

-----Original Message-----
From: Matt Rolf [mailto:rolfm@denison.edu]
Sent: Friday, September 28, 2007 4:02 PM
To: devel@lists.bricolage.cc
Subject: Re: Net::SFTP Slowness

On Sep 28, 2007, at 3:51 PM, David E. Wheeler wrote:

> On Sep 28, 2007, at 11:19, rolfm@denison.edu wrote:
> <rant>
> Have you tried tweaking SFTP_MOVER_CIPHER in bricolage.conf?

No. Might make a difference?

-Matt

Re: Net::SFTP Slowness [ In reply to ]

rolfm at denison

Sep 28, 2007, 1:13 PM

Post #5 of 21 (15727 views)

On Sep 28, 2007, at 4:09 PM, Beaudet, David P. wrote:

> Going from the slowest to the fastest cipher could possibly double or
> triple your speed, but from what it sounded like, you have other
> issues
> going on if you're only getting 100 KB / second transfer rates.
> That's
> a couple orders of magnitude slower than command line SCP over gigabit
> which should be on the order of 20 MB / second.

I played with the ciphers. With Blowfish, it looks like we cut our
time by about 1/3, maybe as much as 40%. That's a big improvement,
but you're right, nothing close to what it should be.

Also, we peg a processor during the whole transfer. It's a big box,
and that's far and away the biggest resource drag.

-Matt

Re: Net::SFTP Slowness [ In reply to ]

rolfm at denison

Oct 15, 2007, 4:22 PM

Post #6 of 21 (15704 views)

Here's a forward from our network engineer showing the difference
between Net::SFTP and Net::SFTP::Foreign.

******
Here are my results with Net::SFTP and Net::SFTP::Foreign scripts
copying a 40 meg file. Both scripts use public-key authentication so
no passwords need to be scripted.

$ time ./sftp0.pl
real 1m29.970s

$ time ./sftp.pl
real 0m1.755s

sftp0.pl uses Net::SFTP.
sftp.pl uses Net::SFTP::Foreign.

The dramatic difference indicates it is worth the pain to convert
Bricolage to using Net::SFTP::Foreign.

******

So we're working on it. The only thing that's stumped us so far is
this step in adding a new distribution method:

* Add an INSERT statement to lib/Bric/Util/Class.val to create a new
representation for your mover class. Be sure to set the value of the
"distributor" column to 1. Use the records for Bric::Util::Trans::FS
and Bric::Util::Trans::FTP as examples.

I'm guessing that's actually in the sql/Pg directory. Is this
directory even included in a distribution?

-Matt

Re: Net::SFTP Slowness [ In reply to ]

david at kineticode

Oct 15, 2007, 8:36 PM

Post #7 of 21 (15713 views)

On Oct 15, 2007, at 17:22, Matt Rolf wrote:

> * Add an INSERT statement to lib/Bric/Util/Class.val to create a
> new representation for your mover class. Be sure to set the value
> of the "distributor" column to 1. Use the records for
> Bric::Util::Trans::FS and Bric::Util::Trans::FTP as examples.
>
> I'm guessing that's actually in the sql/Pg directory. Is this
> directory even included in a distribution?

Yeah, sorry, should be sql/Pg/Bric/Util/Class.val and sql/MySQL/Bric/
Util/Class.val. But why not just change the existing SFTP mover to
use the new module?

Best,

David

Re: Net::SFTP Slowness [ In reply to ]

paulo at digitalcraftsmen

Oct 16, 2007, 12:36 AM

Post #8 of 21 (15730 views)

David E. Wheeler wrote:
> But why not just change the existing SFTP
> mover to use the new module?

I'd definitely go with that. It's such a speed improvement. Also
something like that can be highlighted at the end of the upgrade
process, warning people that they must install this new module.

Paul

Re: Net::SFTP Slowness [ In reply to ]

lannings at who

Oct 16, 2007, 3:36 AM

Post #9 of 21 (15723 views)

On Tue, 16 Oct 2007, Paul Orrock wrote:
> David E. Wheeler wrote:
>> But why not just change the existing SFTP mover to use the new module?
>
> I'd definitely go with that. It's such a speed improvement.

David would normally argue against forking a shell command
(in a mod_perl application). Is that inapplicable here?
(Note: I don't disagree that Net::SFTP::Foreign should be tried,
so don't interpret that too negatively.)

> Also something
> like that can be highlighted at the end of the upgrade process,
> warning people that they must install this new module.

Wouldn't it be normally pointed out when the CPAN dependencies
are checked for? (I assume `make upgrade` does that.)

Re: Net::SFTP Slowness [ In reply to ]

bharder at methodlogic

Oct 16, 2007, 6:36 AM

Post #10 of 21 (15718 views)

On Tue, Oct 16, 2007 at 01:36:26PM +0200, Scott Lanning wrote:
> On Tue, 16 Oct 2007, Paul Orrock wrote:
> >David E. Wheeler wrote:
> >>But why not just change the existing SFTP mover to use the new module?
> >
> >I'd definitely go with that. It's such a speed improvement.

I'm curious why there is such a radical difference in performance. Has the maintainer of the Net::SFTP (David Robbins, if I'm correct) been contacted?

> David would normally argue against forking a shell command
> (in a mod_perl application). Is that inapplicable here?
> (Note: I don't disagree that Net::SFTP::Foreign should be tried,
> so don't interpret that too negatively.)
>
> >Also something
> >like that can be highlighted at the end of the upgrade process,
> >warning people that they must install this new module.
>
> Wouldn't it be normally pointed out when the CPAN dependencies
> are checked for? (I assume `make upgrade` does that.)

--

-bch
http://www.methodlogic.net

Re: Net::SFTP Slowness [ In reply to ]

rolfm at denison

Oct 16, 2007, 6:48 AM

Post #11 of 21 (15715 views)

On Oct 16, 2007, at 10:36 AM, bharder wrote:

> I'm curious why there is such a radical difference in performance.
> Has the maintainer of the Net::SFTP (David Robbins, if I'm
> correct) been contacted?

No, I haven't contacted him. I'll shoot him an e-mail. Net::SFTP
hasn't had any activity for two years, no serious development for 4.
Does that mean it doesn't work right? Well, no. But Robbins does
describe Net::SFTP as legacy near the end of this blog post in February.

http://www.davidrobins.net/?idx=3992

As to the difference in speed, I believe it's the data encoding. The
native client appears to do it much faster than the perl implementation.

And if you don't install Math::BigInt::GMP, creating a connection can
also be slow.

Here's another post describing the massive speed improvement of
Net::SFTP::Foreign.

http://crashingdaily.wordpress.com/2007/09/22/props-to-
netsftpforeigncompat/

-Matt

Re: Net::SFTP Slowness [ In reply to ]

david at kineticode

Oct 16, 2007, 8:53 AM

Post #12 of 21 (15724 views)

On Oct 16, 2007, at 04:36, Scott Lanning wrote:

>> I'd definitely go with that. It's such a speed improvement.
>
> David would normally argue against forking a shell command
> (in a mod_perl application). Is that inapplicable here?
> (Note: I don't disagree that Net::SFTP::Foreign should be tried,
> so don't interpret that too negatively.)

If it shells out to scp or something, then I think I'd prefer to make
it an option -- either a bricolage.conf directive that modifies how
the existing mover works, or a separate mover as Matt originally
described.

Thanks,

David

Re: Net::SFTP Slowness [ In reply to ]

rolfm at denison

Oct 16, 2007, 4:21 PM

Post #13 of 21 (15722 views)

On Oct 16, 2007, at 12:53 PM, David E. Wheeler wrote:

> If it shells out to scp or something, then I think I'd prefer to
> make it an option -- either a bricolage.conf directive that
> modifies how the existing mover works, or a separate mover as Matt
> originally described.

Has anyone developed a wrapper script for testing specific Bricolage
modules? In our case, we want to test Bric::Util::Trans::Foreign. A
certain number of other modules have to be included as well. Our
script fails to compile because of compilation errors which we are
assuming are due to component dependencies. Has anybody else been
able to test some aspect in this way (but successfully)?

We'd also like to capture a Bric::Resource and a Bric::ServerType
that are fully populated. This would allow us to test in a meaningful
way. If not, we'll simply look through the code to determine which
methods are called on the Resource and the ServerType and make sure
those return something expected.

Thanks,

Matt

Re: Net::SFTP Slowness [ In reply to ]

david at kineticode

Oct 16, 2007, 4:35 PM

Post #14 of 21 (15720 views)

On Oct 16, 2007, at 17:21, Matt Rolf wrote:

> We'd also like to capture a Bric::Resource and a Bric::ServerType
> that are fully populated. This would allow us to test in a
> meaningful way. If not, we'll simply look through the code to
> determine which methods are called on the Resource and the
> ServerType and make sure those return something expected.

Check out t/Bric/Util/Trans/Mail/DevTest.pm. That's how all tests
should be written. t/Bric/Util/Trans/Mail/Test.pm, BTW, has tests
that don't touch the database. You should write those when you can,
and DevTest.pm test files when you can't.

HTH,

David

Re: Net::SFTP Slowness [ In reply to ]

rolfm at denison

Oct 16, 2007, 7:16 PM

Post #15 of 21 (15717 views)

Forwarded message from David Robbins on Net::SFTP.

-Matt

Begin forwarded message:

> From: David Robins <dbrobins@davidrobins.net>
> Date: October 16, 2007 10:56:49 PM EDT
> To: Matt Rolf <rolfm@denison.edu>
> Subject: Re: Net::SFTP
>
> I am not planning on further feature development for Net::SFTP, just
> maintenance and bug-fixing.
>
> You may want to look at Net::SSH2, which uses libssh2; both the
> module and the
> library are in active development and are a good deal faster than
> Net::SSH::Perl/Net::SFTP; I know about the performance issues
> there, and as
> you say they're mainly due to the pure-perl implementation;
> libssh2, being C,
> is a good deal faster.
>
>
> --
> Dave
> Isa. 40:31

Re: Net::SFTP Slowness [ In reply to ]

bharder at methodlogic

Oct 16, 2007, 7:41 PM

Post #16 of 21 (15722 views)

On Tue, Oct 16, 2007 at 11:16:05PM -0400, Matt Rolf wrote:
> Forwarded message from David Robbins on Net::SFTP.
>
> -Matt
>
> Begin forwarded message:
>
> >From: David Robins <dbrobins@davidrobins.net>
> >Date: October 16, 2007 10:56:49 PM EDT
> >To: Matt Rolf <rolfm@denison.edu>
> >Subject: Re: Net::SFTP
> >
> >I am not planning on further feature development for Net::SFTP, just
> >maintenance and bug-fixing.
> >
> >You may want to look at Net::SSH2, which uses libssh2; both the
> >module and the
> >library are in active development and are a good deal faster than
> >Net::SSH::Perl/Net::SFTP; I know about the performance issues
> >there, and as
> >you say they're mainly due to the pure-perl implementation;
> >libssh2, being C,
> >is a good deal faster.

Nice... and being a library with Perl bindings, does not have to shell-out to /bin/sftp; /usr/bin/sftp; /usr/local/bin/sftp, or wherever...

> >
> >--
> >Dave
> >Isa. 40:31
>

--

-bch
http://www.methodlogic.net

Re: Net::SFTP Slowness [ In reply to ]

lannings at who

Oct 16, 2007, 11:48 PM

Post #17 of 21 (15734 views)

On Tue, 16 Oct 2007, bharder wrote:
> On Tue, Oct 16, 2007 at 11:16:05PM -0400, Matt Rolf wrote:
>> Forwarded message from David Robbins on Net::SFTP.
>>> You may want to look at Net::SSH2, which uses libssh2;
[...]
> Nice... and being a library with Perl bindings, does not have to shell-out to
> /bin/sftp; /usr/bin/sftp; /usr/local/bin/sftp, or wherever...

ditto

Re: Net::SFTP Slowness [ In reply to ]

rolfm at denison

Jun 9, 2008, 7:11 AM

Post #18 of 21 (15671 views)

On Oct 17, 2007, at 3:48 AM, Scott Lanning wrote:

> On Tue, 16 Oct 2007, bharder wrote:
>> On Tue, Oct 16, 2007 at 11:16:05PM -0400, Matt Rolf wrote:
>>> Forwarded message from David Robbins on Net::SFTP.
>>>> You may want to look at Net::SSH2, which uses libssh2;

Just wanted to post that we are working on making this happen, but it
is taking us longer than we expected. The deletion aspect in
particular is a little tricky. It's made all the more difficult by us
not being able to run Bricolage in debug mode.

However, right now we can put files on the server, so that's
something. Hopefully we can finish this up soon, but I don't know when.

-matt

Re: Net::SFTP Slowness [ In reply to ]

david at kineticode

Jun 9, 2008, 12:48 PM

Post #19 of 21 (15651 views)

On Jun 9, 2008, at 07:11, Matt Rolf wrote:

> Just wanted to post that we are working on making this happen, but
> it is taking us longer than we expected. The deletion aspect in
> particular is a little tricky. It's made all the more difficult by
> us not being able to run Bricolage in debug mode.
>
> However, right now we can put files on the server, so that's
> something. Hopefully we can finish this up soon, but I don't know
> when.

matt++ # Thanks!

David

Re: Net::SFTP Slowness [ In reply to ]

rotoole at gmail

Jun 29, 2009, 5:20 AM

Post #20 of 21 (14729 views)

Little late to this conversation...

I'm experiencing similar slowness with SFTP distribution. Were Matt's
modifications ever rolled into a bric distribution? If not, is changing the
SFTP cipher my best option for speed improvement?

-- - - - - -
-

-
ryan

On Mon, Jun 9, 2008 at 3:48 PM, David E. Wheeler <david@kineticode.com>wrote:

> On Jun 9, 2008, at 07:11, Matt Rolf wrote:
>
> Just wanted to post that we are working on making this happen, but it is
>> taking us longer than we expected. The deletion aspect in particular is a
>> little tricky. It's made all the more difficult by us not being able to run
>> Bricolage in debug mode.
>>
>> However, right now we can put files on the server, so that's something.
>> Hopefully we can finish this up soon, but I don't know when.
>>
>
> matt++ # Thanks!
>
> David
>

Re: Net::SFTP Slowness [ In reply to ]

david at kineticode

Jun 29, 2009, 8:53 AM

Post #21 of 21 (14746 views)

On Jun 29, 2009, at 5:20 AM, Ryan O'Toole wrote:

> I'm experiencing similar slowness with SFTP distribution. Were Matt's
> modifications ever rolled into a bric distribution? If not, is
> changing the
> SFTP cipher my best option for speed improvement?

They're in 1.11.1, yes. It's a dev release, but quite a few people are
using it in production.

Best,

David