My Perl/Expect script aborts when sftp connection stalls at remote destination

c84032002 has asked for the wisdom of the Perl Monks concerning the following question:

I'm using Expect to automate an sftp session with a site that only uses passwords, not ssh keys. It worked fine until it tried retrieving an oversized file. The session log indicated the connection "stalled" too much traffic & users on remote site and my script terminated. I tried upping the timeout parm in the comand '$exp->expect(600, "sftp>");' that's waiting for the sftp prompt once the file get is done, but it didn't help. It bombed well before the 10 minutes passed. Since I can expect the transmission to stall almost every time, I can't just use a loop to check the session log and try again. Is there anything deeper in Perl or Expect that I could use to help my script cope with stalled transmissions? Thanks, in advance, for any ideas or suggestions. ------------------------------ at the risk of violating a posting rule on over-bloated questions, here is the script. It reads in a file of the actual sftp commands but they are just vanilla puts or gets mostly. ------------

use strict;
use Expect;

my $sftpPassword = "useracctPW";
open FTP_CARDS, "$ARGV[0]/sftp_CMS.cards" or die $!;

# Uncomment the following line if you want to see what expect is doing
# $Expect::Exp_Internal = 1;

# Create the Expect object and create the message log file.
my $params = " -oPort=10022 useracct\@sftp.section111.cms.hhs.gov";
my $command = "/usr/bin/sftp $params";
my $exp = Expect->spawn("$command") or die "Cannot spawn sftp command 
+\n";

$exp->timeout(600);

## [this does not append ]$exp->log_file("$ARGV[0]/sftp_CMS.log", "w")
+;
$exp->log_file("$ARGV[0]/sftp_CMS.log");
$exp->log_stdout(0);

# Wait for Password prompt to show up then send it back
$exp->expect(30, ["Password:"]);
## $exp->expect(30, ["useracct\@sftp.section111.cms.hhs.gov's password
+:"]);
$exp->send("$sftpPassword\n");

# Execute each sftp command found in the input file of commands
while (my $ftp_command = <FTP_CARDS>) {
   # Wait for sftp prompt
   $exp->expect(600, ["sftp>"]);
   $exp->send("$ftp_command\n");
}

# Destroy the expect object
$exp->soft_close();
[download]

Comment on My Perl/Expect script aborts when sftp connection stalls at remote destination Download Code

Replies are listed 'Best First'.
Re: My Perl/Expect script aborts when sftp connection stalls at remote destination by zow (Sexton) on Feb 24, 2010 at 19:24 UTC
Does the hang-up occur when running the sftp commands on the command line? You could also try Net::SFTP::Foreign or Net::SFTP (lots of dependencies for this one). I find working with those packages to be a little better with respect to exception handling for doing SFTP.	[reply]
Re^2: My Perl/Expect script aborts when sftp connection stalls at remote destination by Anonymous Monk on Feb 24, 2010 at 19:52 UTC
First, zow, thanks for the quick reply. When connected manually I see the 1-line status info line progressing through from 0% to 100%, with the sporadic message of stalled. But it finishes. For the script's run it seems to just cut off, maybe at the first stall, as if I did a kind of "kill" command from the UNIX command line. As to switching to Net::SFTP, that'd be a tough switch since our script is already in production and we don't have Net::SFTP in our shop at this point. I know it sounds dumb to have a frail app in production. Most future files are expected to be small enough that the problem vanishes, and I can manually get the rare files that are too big. ALSO, I'll see if a late night run of the script, when the traffic should be much lower, avoids the problem.	[reply]
Re^3: My Perl/Expect script aborts when sftp connection stalls at remote destination by bobthebaritone (Sexton) on Feb 25, 2010 at 10:17 UTC
Is there any reason not to use the complete expect to run this? The PERL module is not a fully implementation of expect as far as I know. I recall from Don Libes book ( the writer of expect) that there is excellent handling of exceptions you speak of in native expect. I'd be interested to give this some more thought as I use native expect in a PERL shop. I have been thinking of using the module, but found that Expect is a complete language!	[reply]
Re^3: My Perl/Expect script aborts when sftp connection stalls at remote destination by salva (Canon) on Feb 25, 2010 at 15:09 UTC
Really, forget about solving your problem with Expect and use one of the Perl SFTP modules. It doesn't matter how hard it is to get it or a new version of the script in production. Properly done, your file transfers will be reliable and you will not have to worry about them anymore. It will pay back! My own recommendation is to go with Net::SFTP::Foreign, but I am the author, not completely unbiased!	[reply]
Re^3: My Perl/Expect script aborts when sftp connection stalls at remote destination by zow (Sexton) on Feb 25, 2010 at 19:26 UTC
One other thing that I noticed is that in your while loop over your commands you are calling the expect method with the 600 sec timeout before the send method. Perhaps it is still using a default timeout?	[reply]
Re: My Perl/Expect script aborts when sftp connection stalls at remote destination by zentara (Cardinal) on Feb 25, 2010 at 14:11 UTC
You might want to look at Net::ssh2 timeout and look at syphilis's reply about setting the poll(0). Net::SSH2 does sftp. Possibly, you could find how to set the poll() from your script. Upon looking thru the Net::SSH2 docs for clues, it seems a timeout value for the connection gets set `connect ( handle \| host [, port [, Timeout => secs ]] )` [download] Since sftp uses ssh2 libs, there probably is some arcane way of you setting the ssh2 options from sftp. Or do like the rest of the modern world, and move up to Net::SSH2. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku	[reply] [d/l]
Re^2: My Perl/Expect script aborts when sftp connection stalls at remote destination by salva (Canon) on Feb 25, 2010 at 14:58 UTC
Net::SSH2 does sftp Only in a rather limited way! Net::SSH2 implements the low level methods defined in the SFTP draft but lacks the most useful methods you could expect from an SFTP client (i.e. get or put). Writting these methods yourself from the protocol primitives in an efficient manner is very hard.	[reply]