Faile has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

Is there a way to regulate how sockets are used by Net::FTP?

To me it sounds like there could be a way to "reuse" sockets since the transfers do not happen in parallell but sequentially.

Situation; I have a dropbox server from which I copy the incoming files to an internal server using a perl script. The script also archives the files and makes a copy to a dev environment on demand and writes these transfers to a database for tracking.

The problem I'm facing is that given hundreds of files every 15 minutes means that the script opens up a truckload of sockets ( nearly 1500 simultaneous open sockets ) for all the put commands.

Here are the FTP specific commands used in the script.

This part is exectuted only once:

$ftp2=Net::FTP->new($host2,Port=>$port,Timeout=>60) or die "Cannot con
+nect2 to $host2: $@";
$ftp2->login($user2,$pass2) or die "ftp2: Cannot login ", $ftp->messag
+e;
$ftp2->binary or die "Couldn't change mode2 to binary!\n";

[download]

This code is executed for every file encountered that should be moved, finding the files to be moved is done using File::Find. In the "wanted" loop I call on the archival of the file and then the transfer:

$ftp2->put($file) or my $problem2 = 1;

[download]

If there's no way to limit the sockets easily, I would appreciate if you could give me pointers to better ways to do this. I used File::Copy earlier to copy the files over SMB to UNC paths but found the solution to be rather unstable, causing intermittent errors due to unknown reasons ( probably network related ).

Thanks in advance

Replies are listed 'Best First'.
Re: limiting the amount of sockets opened by f
by ikegami (Patriarch) on Feb 10, 2012 at 08:09 UTC

    To me it sounds like there could be a way to "reuse" sockets since the transfers do not happen in parallell but sequentially.

    Each FTP transfer (including transfers of directory listings) uses a new connection. Net::FTP cannot act differently.

    That said, it shouldn't have more than two sockets open at a time. If it has 1,500 open sockets, something isn't right. Or do you mean sockets in the TIME_WAIT state?

[reply]

      Regarding TIME_WAIT, you got me there, I didn't actually suspect they might be closed and timing out, is this normal behaviour? Are there any ways for me to report number of sockets being kept open within perl?

      This problem report came to me from a sysadmin complaining to me that when my script runs, there's tons of sockets everywhere and other processes are acting strangely, like an ftp server being intermittently unresponsive .

[reply]

        Regarding TIME_WAIT, you got me there, I didn't actually suspect they might be closed and timing out, is this normal behaviour?

        Yes. A TCP port is marked as unavailable (TIME_WAIT) for an amount of time after being closed to avoid having a delayed packet from one connection affect a later connection on the same port. This can be disabled on a per-socket basis (at least).

        Are there any ways for me to report number of sockets being kept open within perl?

        I sure there is, but I don't know what tools are available.

        On some systems, you could peek into /proc/$$/fd/ ($$ representing the process's PID). You'll see open file handles, which includes open sockets. 

        $ perl -MIO::Socket::INET -E'
   system "ls -lF /proc/$$/fd/";
   my $s = IO::Socket::INET->new("www.google.com:80") or die $!;
   system "ls -lF /proc/$$/fd/";
'
total 0
lrwx------ 1 eric users 64 Feb 14 15:22 0 -> /dev/pts/7
lrwx------ 1 eric users 64 Feb 14 15:22 1 -> /dev/pts/7
lrwx------ 1 eric users 64 Feb 14 15:22 2 -> /dev/pts/7
lr-x------ 1 eric users 64 Feb 14 15:22 3 -> pipe:[2358130]
total 0
lrwx------ 1 eric users 64 Feb 14 15:22 0 -> /dev/pts/7
lrwx------ 1 eric users 64 Feb 14 15:22 1 -> /dev/pts/7
lrwx------ 1 eric users 64 Feb 14 15:22 2 -> /dev/pts/7
lr-x------ 1 eric users 64 Feb 14 15:22 3 -> socket:[2358143]
lr-x------ 1 eric users 64 Feb 14 15:22 4 -> pipe:[2358144]

        [download]

        (0,1,2 = std*. The pipe is used by system to communicate exec failures to the parent.)

        Update: netstat on my machine has a -p option that shows the PID of the process that owns the socket.

[reply]
[d/l]
[select]
Re: limiting the amount of sockets opened by f
by rovf (Priest) on Feb 10, 2012 at 09:27 UTC
    Could it be that you forgot to close the socket? ($ftp2->quit)

    -- 
    Ronald Fischer <ynnor@mm.st>
[reply]
[d/l]
[select]

      Thanks for the reply, I do call ftp->quit but that will close the whole session so it's done after all the files are transferred.

      If you look at FTP.pm you'll see that sub put actually does always call $sock->close after a transfer is done, so no sockets should be "left over"

      Regarding TIME_WAIT, you got me there, I didn't actually suspect they might be closed and timing out, is this normal behaviour? Are there any ways for me to report number of sockets being kept open within perl?

      This problem report came to me from a sysadmin complaining to me that when my script runs, there's tons of sockets everywhere and other processes are acting strangely, like an ftp server being unresponsive intermittently.

[reply]

Back to Seekers of Perl Wisdom