SheridanCat has asked for the wisdom of the Perl Monks concerning the following question:

It's a puzzle for me, at least. I have a relatively simple script that uploads jpeg files from a local host to a SAN in a data center. I'm using Net::SFTP::Foreign to get the job done and it does a good job, except that it starts failing on the 100th file uploaded - every time. The code is below. I've anonymized it and removed a bunch of logging statements.
#!/usr/bin/perl use warnings; use strict; use Data::Dumper; use Getopt::Long; use Pod::Usage; use Net::SFTP::Foreign::Compat; use File::Find; use File::Basename; use vars qw( $opt_verbose $opt_path $opt_type @queue ); GetOptions( "path=s" => \$opt_path, "type|t=s" => \$opt_type, "verbose|v" => \$opt_verbose, ); my $remoteh = get_uploader( debug => $opt_verbose, host => 'remote_host', port => 5005, user => 'username', priv_key => '/root/content_upload/.ssh/p +rivate.ppk', ); my $target_root = 'remote05-0/artwork' find( { wanted => \&found, follow => 1 }, $opt_path ); my $count = 0; my $to; foreach my $file ( @queue ){ $count++; print $count, "\t", $file, "\n"; # Get the filename and use only the first 3 digits to get a subdirec +tory name my $basename = basename( $file, '.jpg' ); $basename =~ m/(\d{3})/; my $remote_dir = $1; # Build up the full path and name of the file for the remote system + $to = join '/', $target_root, $remote_dir, $basename . '.jpg'; # NOTE: The do_opendir() and do_mkdir() correspond to the line 67 & +68 errors in the output # See if the subdir exists on the remote system and, if not, create +it my $check_path = $remoteh->do_opendir( $target_root . '/' . $remote_ +dir ); if( ! $check_path ){ $remoteh->do_mkdir( $target_root . '/' . $remote_dir, Net::SFTP::F +oreign::Attributes->new() ); } # NOTE: The put() below corresponds to the Line 77 errors in the out +put # Copy it up my $copy_status = $remoteh->put( $file, $to ); } sub found{ # We only want jpegs starting with a digit return unless m/\.jpg$/i; return unless m/^\d/; # put the entire path/filename in an array push @queue, $File::Find::name; } sub get_uploader{ my ( %args ) = @_; my %ssh_args = ( identity_file => $args{'priv_key'}, ); my %sftp_args = ( user => $args{'user'}, port => $args{'port'}, debug => $args{'debug'}, ssh_args => \%ssh_args, ); return Net::SFTP::Foreign::Compat->new( $args{'host'}, %sftp_args ); }
And here's the error output. One thing to note, these errors are from the non-anonymized code, so the line numbers in the errors are not directly relatable to the code above. I've noted in comments above where they correspond.
95 /cpnfs/image/scans/032431017364.jpg 96 /cpnfs/image/scans/032431017463.jpg 97 /cpnfs/image/scans/032431017562.jpg 98 /cpnfs/image/scans/032431017661.jpg 99 /cpnfs/image/scans/032431017760.jpg 100 /cpnfs/image/scans/032431017869.jpg Couldn't get handle: Failure at ./image_upload.pl line 77 101 /cpnfs/image/scans/032431018064.jpg Couldn't get handle: Failure at ./image_upload.pl line 67 Couldn't create directory: Failure at ./image_upload.pl line 68 Couldn't get handle: Failure at ./image_upload.pl line 77 102 /cpnfs/image/scans/032431018163.jpg Couldn't get handle: Failure at ./image_upload.pl line 67 Couldn't create directory: Failure at ./image_upload.pl line 68 Couldn't get handle: Failure at ./image_upload.pl line 77 103 /cpnfs/image/scans/032431018262.jpg Couldn't get handle: Failure at ./image_upload.pl line 67 Couldn't create directory: Failure at ./image_upload.pl line 68 Couldn't get handle: Failure at ./image_upload.pl line 77
I'm guessing I'm filling a buffer, but I'm not sure where or how to avoid it. If I have to I can blow away the SFTP handle and reconnect, but that seems wasteful. Also, I don't see a disconnect() method, so I guess I would have to just undef the handle. Thanks for any guidance you might be able to give.

Replies are listed 'Best First'.
Re: Net::SFTP::Foreign Puzzle
by Hue-Bond (Priest) on Jul 06, 2006 at 16:18 UTC
    my $check_path = $remoteh->do_opendir( $target_root . '/' . $remote_di +r ); Couldn't get handle: Failure at ./image_upload.pl line 77

    Wild guess: are you closing the opened filehandles?

    --
    David Serrano

      Exactly right. I made this change:
      my $check_path = $remoteh->do_opendir( $target_root . '/' . $remote_di +r ); if( ! $check_path ){ $remoteh->do_mkdir( $target_root . '/' . $remote_dir, Net::SFTP::For +eign::Attributes->new() ); }else{ $remoteh->do_close( $check_path ); }
      Now all is well. Thank you so much for pointing that out. Very dumb of me.

        You are still leaving opened filehandles, the ones for which !$check_path is true. I would close the filehandles unconditionally just after the $remoteh->put line.

        --
        David Serrano

Re: Net::SFTP::Foreign Puzzle
by salva (Canon) on Jul 07, 2006 at 06:48 UTC
    You should try the development version of Net::SFTP::Foreign that has a better API. Specifically, remote file and directory handles are automatically closed when they go out of scope.

    BTW, you don't really need to check if the remote directory exist, just try to create it and the command will fail (doing nothing) if it is already there.

      Thanks, salva. That's the behavior I was counting on with the current implementation. I'll check it out.