Net::SSH2 fails under heavy load

FloydATC has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to copy thousands of files using Net::SSH2 (tried Net::SSH2::Perl before, see http://www.perlmonks.org/index.pl?node_id=765848 )

So far, I've tried three different approaches, none of them successful:

my $sftp = $ssh2->sftp
Fails after a few dozen, hundred or thousand iterations (the exact number varies greatly)

libssh2_sftp_init(ss->session) -> 0x9a641d8
(OK, proceed with file copy)
libssh2_sftp_open_ex(sf->sftp, (char*)pv_file, len_file, l_flags, mode
+, 0) -> 0x9ce1dd8
Net::SSH2::File::DESTROY
Net::SSH2::SFTP::DESTROY
Net::SSH2::SFTP::DESTROY freeing session
libssh2_sftp_init(ss->session) -> 0x9a641d8
(OK, proceed with file copy)
libssh2_sftp_open_ex(sf->sftp, (char*)pv_file, len_file, l_flags, mode
+, 0) -> 0x9ce0900
Net::SSH2::File::DESTROY
Net::SSH2::SFTP::DESTROY
Net::SSH2::SFTP::DESTROY freeing session
libssh2_sftp_init(ss->session) -> 0x0
(FAIL)
Net::SSH2::Channel::DESTROY
Net::SSH2::DESTROY object 0x95c4390
[download]

The comments in parenthesis are mine, notice that the same object handle is returned with each successful call until it suddenly returns 0x0

Second attempt:
$ssh2->scp_get($old, $new)
Hangs silently after a few dozen, hundred or thousand iterations (the exact number varies greatly)

Net::SSH2::Channel::read(size = 8192, ext = 0)
- read 8192 bytes
- read 8192 total
Net::SSH2::Channel::read(size = 8192, ext = 0)
- read 4899 bytes
- read 3293 bytes
- read 8192 total
Net::SSH2::Channel::read(size = 8192, ext = 0)
- read 8192 bytes
- read 8192 total
Net::SSH2::Channel::read(size = 8192, ext = 0)
- read 7228 bytes
[download]

...and it just sits there, waiting for the cows to come home.

Finally I try

my $chan = $ssh2->channel();
$chan->blocking(1);
$chan->exec("cat myfile");
[download]

with results quite similar to sftp:

libssh2_channel_open_ex(ss->session, pv_channel_type, len_channel_type
+, window_size, packet_size, ((void *)0) , 0 ) -> 0x916e480
 [53819:0]Net::SSH2::Channel::read(size = 1048576, ext = 0)
- read 16384 bytes
- read 16384 bytes
- read 16384 bytes
- read 4667 bytes
- read -37 bytes
- read 53819 total
Net::SSH2::Channel::DESTROY
libssh2_channel_open_ex(ss->session, pv_channel_type, len_channel_type
+, window_size, packet_size, ((void *)0) , 0 ) -> 0x0
[download]

Channel creation fails, three strikes and I'm out.

Comment on Net::SSH2 fails under heavy load Select or Download Code

Replies are listed 'Best First'.
Re: Net::SSH2 fails under heavy load by salva (Canon) on May 25, 2009 at 07:42 UTC
Net::SSH2 is still a work in progress... For SFTP you can use Net::SFTP::Foreign that is more mature and feature rich.	[reply]
Re^2: Net::SSH2 fails under heavy load by FloydATC (Deacon) on May 25, 2009 at 08:50 UTC
Thanks for the tip, I'll have a look at it.	[reply]
Re: Net::SSH2 fails under heavy load by syphilis (Archbishop) on May 25, 2009 at 08:06 UTC
The ouptut you're getting with `$ssh2->scp_get($old, $new)` differs quite markedly from what I get. I'm running with `$ssh2->debug(1);`, and I gather you're doing the same ? My particular code segment, which is doing the job fine, is as follows: `for(@files) { $ret = $ssh2->scp_get("$dir/$_", "$_"); print "RET: $ret\n"; }` [download] My `@files` contains 6 filenames, and the output that segment produces is: libssh2_scp_recv(ss->session, path, &st) -> 0x3dae38c Net::SSH2::Channel::read(size = 506, ext = 0) - read 506 bytes - read 506 total Net::SSH2::Channel::read(size = 1, ext = 0) - read 1 bytes - read 1 total Net::SSH2::Channel::DESTROY RET: 1 libssh2_scp_recv(ss->session, path, &st) -> 0x3dae38c Net::SSH2::Channel::read(size = 500, ext = 0) - read 500 bytes - read 500 total Net::SSH2::Channel::read(size = 1, ext = 0) - read 1 bytes - read 1 total Net::SSH2::Channel::DESTROY RET: 1 libssh2_scp_recv(ss->session, path, &st) -> 0x3dae38c Net::SSH2::Channel::read(size = 499, ext = 0) - read 499 bytes - read 499 total Net::SSH2::Channel::read(size = 1, ext = 0) - read 1 bytes - read 1 total Net::SSH2::Channel::DESTROY RET: 1 libssh2_scp_recv(ss->session, path, &st) -> 0x3dae38c Net::SSH2::Channel::read(size = 507, ext = 0) - read 507 bytes - read 507 total Net::SSH2::Channel::read(size = 1, ext = 0) - read 1 bytes - read 1 total Net::SSH2::Channel::DESTROY RET: 1 libssh2_scp_recv(ss->session, path, &st) -> 0x3dae38c Net::SSH2::Channel::read(size = 570, ext = 0) - read 570 bytes - read 570 total Net::SSH2::Channel::read(size = 1, ext = 0) - read 1 bytes - read 1 total Net::SSH2::Channel::DESTROY RET: 1 libssh2_scp_recv(ss->session, path, &st) -> 0x3dae38c Net::SSH2::Channel::read(size = 493, ext = 0) - read 493 bytes - read 493 total Net::SSH2::Channel::read(size = 1, ext = 0) - read 1 bytes - read 1 total Net::SSH2::Channel::DESTROY RET: 1 [download] I'm thinking it's the `Net::SSH2::Channel::DESTROY` that occurs between successive scp_get() calls that might be critical. I'm using Net-SSH2-0.19 (built against libssh2-1.0). I haven't tried grabbing large numbers of files - and if I did, then faik I might well strike exactly the same problem as you. Cheers, Rob	[reply] [d/l] [select]
Re^2: Net::SSH2 fails under heavy load by FloydATC (Deacon) on May 25, 2009 at 09:10 UTC
I get similar results most of the time. It appears to fail quite randomly though, sometimes the channel setup fails and sometimes the read() just hangs silently. Hangs seem to happen more often with scp_get(). The channel/exec method is most granular and so I've managed to get it sort-of-working by re-connecting and re-opening whenever things fail. When this job is done, the code needs to be destroyed for the sake of humanity.	[reply]