Cody Pendant has asked for the wisdom of the Perl Monks concerning the following question:

I've got a script which parses some spreadsheet data and uploads it to a web server using Net::FTP.

In fact it uploads twice, once to an internal server (where it's checked) and then to an external server, if the user presses a "continue, everything's fine" button.

The internal server is LINUX and Net::FTP, in normal FTP mode, uploads in a second. The external server is a supposedly-very-robust "data appliance" which uses NT to manage FTP.

I can't even FTP to this external server in normal FTP mode, I have to use Passive Mode:

$ftp = Net::FTP->new("external.host.name", Passive => 1);

And this is all very well, but the actual file transfer (10Kb at worst) seems to take ages. The logon is instant, the CWD is instant, the rename of the old file with ".bak" is instant, but then the actual upload appears to take something like three or four minutes.

A desktop FTP client upload is also more or less instant for the same file.

Can any monks shed light on this for me please? To be frank, I don't really know what Passive FTP means, or why I can't use regular FTP for this server.

Is the transfer really taking three minutes, or is it somehow being queued by the server and accepted after three minutes, or is it happening instantly and the server taking three minutes to report success?

Not that I care which answer is correct if you can help me speed it up.



($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss')
=~y~b-v~a-z~s; print

Replies are listed 'Best First'.
Re: Net::FTP takes a very long time in Passive mode
by hsinclai (Deacon) on Oct 16, 2004 at 02:21 UTC
    I can't even FTP to this external server in normal FTP mode, I have to use Passive Mode:
    To be expected, if the external server is behind a firewall not allowing a standard ftp connection..

    If this is true
    instant, but then the actual upload appears to take something like three or four minutes.

    and this is true
    A desktop FTP client upload is also more or less instant for the same file.

    then you are making the connection and transferring the file. Therefore passive/active is certainly not the problem.

    I am guessing your problem is one or a combination of TCP window sizing (mismatches across the transit interfaces), or MTU/sizing or path MTU discovery enabling/disabling.

    Here's why:
      Thanks so much for all that detail. I'm sorry to have to tell you that the machine doing the transfer to the Windows machine isn't a LINUX box, it's actually Mac OS X. There are actually three webservers involved.

      The process is:

      1. Script on OS X webserver uploads and parses files.
      2. Script on OS X webserver FTPs to internal LINUX box (all goes well)
      3. Script on OS X webserver FTPs to NT/Firewall box (long delay)

      So I need to find out how to tweak those values on the Mac server.



      ($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss')
      =~y~b-v~a-z~s; print
        Ah.

        Well on OSX that should be done with sysctl, which should be fairly analogous to the old FreeBSD sysctl. There are less values than Linux but you can set these at least:
        net.inet.tcp.sendspace: 32768 + net.inet.tcp.recvspace: 57344 + net.inet.tcp.path_mtu_discovery: 1

        You should be able to use (as root) this form
        sysctl -w net.inet.tcp.recvspace=40960

        or in /etc/sysctl.conf set the value with this form
        net.inet.tcp.recvspace=40960
        so any values are maintained across reboots.

        So I suspect the thing to look for is your default send window being too large. If that was the case the delay could be caused by excessive packet reassembly problems, with more hops to the NT probably making it worse. I would also look on the NT box at the firewall logs when you connect .. PS I don't know how close those commands are but it should get you close..
Re: Net::FTP takes a very long time in Passive mode
by pg (Canon) on Oct 16, 2004 at 00:05 UTC

    If I didn't remember wrong... for FTP, there are two communication channels used, one is control channel, always activated by the client, means the client connects to the server. The other one is data channel, for data channel, you can either act as server (waiting for connection, thi sis the normal way, and you tell the server the port you are listening by sending PORT command), or you reqire the server to listen, and you connect to it (this is called passive mode).

    Update: (copied from RFC 959)

    control connection The communication path between the USER-PI and SERVER-PI for the exchange of commands and replies. This connection follow +s the Telnet Protocol. data connection A full duplex connection over which data is transferred, in a specified mode and type. The data transferred may be a part o +f a file, an entire file or a number of files. The path may be between a server-DTP and a user-DTP, or between two server-DTPs. data port The passive data transfer process "listens" on the data port for a connection from the active transfer process in order to open the data connection.
Re: Net::FTP takes a very long time in Passive mode
by BrowserUk (Patriarch) on Oct 16, 2004 at 00:34 UTC
    To be frank, I don't really know what Passive FTP means, or why I can't use regular FTP for this server.

    See ikegami's excellent post at FTP Connections and Firewalls.


    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon

      In that post, twice the author said listening on a random port, it is correct to a level...

      1. Both the user and the server-DTPs have a default data port. The user-process default data port is the same as the control connection port (i.e., U). The server-process default data port is the port adjacent to the control connection port (i.e., L-1).
      2. Default Data Connection Ports: All FTP implementations must support use of the default data connection ports, and only the User-PI may initiate the use of non-default ports.
      3. Negotiating Non-Default Data Ports: The User-PI may specify a non-default user side data port with the PORT command. The User-PI may request the server side to identify a non-default server side data port with the PASV command.

        It would make more sense to post your correction as a reply to the post that it is correcting?


        Examine what is said, not who speaks.
        "Efficiency is intelligent laziness." -David Dunham
        "Think for yourself!" - Abigail
        "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon