bowei_99 has asked for the wisdom of the Perl Monks concerning the following question:

It seems the Net::FTP::Recursive object is having trouble with closing the ftp connection. That said, as it seems to be about to do a listing of directories, I'm not sure why it would closing it without giving the output of the listing.

I run my script, which errors out at line 54 in one of the module files:

[root@host6 dell-download]# ./get-files.pl > out.log 2>&1 & [1] 10027 [root@host6 dell-download]# tail -f out.log Net::FTP::Recursive=GLOB(0x1c109a8)>>> CDUP Net::FTP::Recursive=GLOB(0x1c109a8)<<< 250 CDUP command successful. Net::FTP::Recursive=GLOB(0x1c109a8)>>> CWD Red Hat Enterprise Linux 4. +7 Net::FTP::Recursive=GLOB(0x1c109a8)<<< 250 CWD command successful. Making dir: Red Hat Enterprise Linux 4.7 Calling rget in /Browse_For_Drivers/Servers, Storage & Networking/Powe +rEdge/PowerEdge R610/Network/HTML Net::FTP::Recursive=GLOB(0x1c109a8)>>> PASV Net::FTP::Recursive=GLOB(0x1c109a8)<<< 227 Entering Passive Mode (143, +166,135,12,206,55) Net::FTP::Recursive=GLOB(0x1c109a8)>>> LIST Can't use an undefined value as a symbol reference at /usr/share/perl5 +/Net/FTP/dataconn.pm line 54. ^C [1]+ Exit 255 ./get-files.pl > out.log 2>&1
I can ftp to the server manually and get a listing just fine:
ftp> cd "Browse_For_Drivers/Servers, Storage & Networking/PowerEdge/Po +werEdge R610/Network/HTML" 250 CWD command successful. ftp> ls 227 Entering Passive Mode (143,166,135,12,221,177) 125 Data connection already open; Transfer starting. drwxrwxrwx 1 owner group 0 Aug 29 2012 Linux drwxrwxrwx 1 owner group 0 Aug 29 2012 LINUX - OS ..<listings removed>.. drwxrwxrwx 1 owner group 0 Aug 29 2012 Windows Ser +ver 2008 x86 226 Transfer complete.
I open /usr/share/perl5/Net/FTP/dataconn.pm, which shows (line numbers also shown):
48 sub _close { 49 my $data = shift; 50 my $ftp = ${*$data}{'net_ftp_cmd'}; 51 52 $data->SUPER::close(); 53 54 delete ${*$ftp}{'net_ftp_dataconn'} 55 if exists ${*$ftp}{'net_ftp_dataconn'} 56 && $data == ${*$ftp}{'net_ftp_dataconn'}; 57 }
This _close function is called here:
60 sub close { 61 my $data = shift; 62 my $ftp = ${*$data}{'net_ftp_cmd'}; 63 64 if (exists ${*$data}{'net_ftp_bytesread'} && !${*$data}{'net +_ftp_eof'}) { 65 my $junk; 66 $data->read($junk, 1, 0); 67 return $data->abort unless ${*$data}{'net_ftp_eof'}; 68 } 69 70 $data->_close;
I checked with our network admin, who said the inactivity timeout is one hour, i.e. if there is no data being sent in a 1 hour period, the firewall will close the connection. I'd think there's probably something similar on the ftp server side as well. However, I can't see, from the debug output above, why a long period of inactivity would happen. As I mentioned, it seemed like it was about to do a directory listing, then got cut off.

Below is the code in question. While I'm creating my own directory structure, I'm creating an ftp connection for each directory and changing directories each time, so I don't see why this should be causing this problem.

#!/usr/bin/perl use strict; use warnings; use Carp; use Data::Dumper; use Net::hostent; #use Net::Ping; use Net::FTP::Recursive; use Log::StdLog { level => 'info', file => $0 . "log" }; #use Log::StdLog { level => 'warn', file => $config{log}{file} . $0 . +"log" }; my %params = ( site => "ftp.dell.com", basedir => "Browse_For_Drivers/Servers, Storage & Networking/Power +Edge", modeldirs => [ "PowerEdge R810", "PowerEdge R610", "PowerEdge R720", "PowerEdge R620", "PowerEdge M620", "PowerEdge M1000E", ], ); print {*STDLOG} info => "Starting $0, creating dirs."; for my $dir (@{ $params{modeldirs}}) { mkdir $dir; chdir $dir; FTPConnect (\%params, "$dir"); chdir ".."; } print {*STDLOG} info => "Finished $0."; sub FTPConnect { my $ref_params = shift @_; my $dir = shift @_; my $ftp = Net::FTP::Recursive->new($ref_params->{site}, Debug => 1 +, Timeout => 15); if ($ftp) { print {*STDLOG} debug => "OK: connected via FTP to " . $ref_p +arams->{site} ; $ftp->login("anonymous",'me@here.there'); $ftp->binary; $ftp->cwd($ref_params->{basedir} . '/' . $dir); print {*STDLOG} info => "Starting download for dir $dir.\n"; $ftp->rget( #FlattenTree => 1, MatchFiles => qr/\.txt$/, ); print {*STDLOG} info => "Finished download for dir $dir.\n"; $ftp->quit; } else { print {*STDLOG} warn => "ERROR: FTP for host $ref_params->{site}\n +" } return ; } sub GetFiles { return; }
Any thoughts on why the connection dying? It's happened a number of times already.

-- Burvil

Replies are listed 'Best First'.
Re: Net::FTP::Recursive problems, prematurely closing connection
by roboticus (Chancellor) on Dec 16, 2013 at 22:22 UTC

    bowei_99:

    Why would you think there's a networking issue? The error message clearly states that a data value isn't present. You may want to run your code under the debugger and monitor the variable(s) of interest at different locations of your program until you find where the error is.

    By the way, I've not seen anyone use an asterisk in a hash key like that. What do you suppose that is doing? Amusing, I don't recall seeing people intentionally use the stringified reference for a hash key!

    Finally, if your issue is that you don't know why the code is trying to close the connection, then it's not terribly helpful to see only the closing logic. By that point, the decision to close has *already been made*, and you're just showing us what's being executed after the incorrect decision.

    Since you've added your code to your post, I looked it over, and I don't see any obvious problems. Are you running it directly from the command line? If not, perhaps a helpful error message is getting swallowed somewhere and not showing up.

    Update: I've added the italicized bits after the OP added more code to his post.

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

      I've not seen anyone use an asterisk in a hash key like that. What do you suppose that is doing?
      In fairness that code is from Net::FTP::dataconn.pm so I don't think that's the problem.
Re: Net::FTP::Recursive problems, prematurely closing connection
by tangent (Parson) on Dec 17, 2013 at 00:41 UTC
    Just so you know, I tried your updated code and it ran successfully producing the below structure on my local hard drive (part listing). So perhaps your problem lies elsewhere, maybe with the firewall you mention.
    PowerEdge R610 BIOS Windows Server 2008 x86 BIOS OS Independent 2009-03-30 - R610-010004BIOS.txt 2009-05-27 - R610-010104BIOS.txt 2009-08-31 - R610-010206BIOS.txt 2009-12-14 - R610-010306BIOS.txt 2010-04-15 - R610-020013BIOS.txt ... Chipset Driver Windows Server 2003 Windows Server 2003 x64 Windows Server 2008 R2 Windows Server 2008 x64
Re: Net::FTP::Recursive problems, prematurely closing connection
by tangent (Parson) on Dec 16, 2013 at 23:03 UTC
    I tried this and didn't encounter the same problem - output:
    ... Net::FTP::Recursive=GLOB(0x8af2a0)>>> CDUP Net::FTP::Recursive=GLOB(0x8af2a0)<<< 250 CDUP command successful. Net::FTP::Recursive=GLOB(0x8af2a0)>>> CWD Red Hat Enterprise Linux 4.7 Net::FTP::Recursive=GLOB(0x8af2a0)<<< 250 CWD command successful. Calling rget in /Browse_For_Drivers/Servers, Storage & Networking/Powe +rEdge/PowerEdge R610/Network/HTML Net::FTP::Recursive=GLOB(0x8af2a0)>>> PASV Net::FTP::Recursive=GLOB(0x8af2a0)<<< 227 Entering Passive Mode (143,1 +66,147,12,243,69) Net::FTP::Recursive=GLOB(0x8af2a0)>>> LIST Net::FTP::Recursive=GLOB(0x8af2a0)<<< 150 Opening BINARY mode data con +nection. Net::FTP::Recursive=GLOB(0x8af2a0)<<< 226 Transfer complete. -rwxrwxrwx 1 owner group 7582757 Sep 6 2012 2009-04-02 +- Intel_LAN_11.0_Manual_A00.zip -rwxrwxrwx 1 owner group 7659340 Oct 28 2009 2009-09-28 +- Intel_LAN_11.5_Manual_A01.zip Net::FTP::Recursive=GLOB(0x8af2a0)>>> PWD ...
    What perplexes me is this line in your output:
    Making dir: Red Hat Enterprise Linux 4.7
    Why is it trying to make a directory? Can you post your own relevant code?

      I think Net:FTP:Recursive is creating a local mirror directory structure. That is one of its modes of operation.

      It looks like Net::FTP:Recursive will take a debug option that may provide more info. It inherits this from Net::FTP.

      Net::FTP::Recursive->new("some.host.name", Debug => 1);
      I have my own code that's trying to create a local directory structure; I'll post the code in the main body.

      -- Burvil