in reply to Net::FTP takes a very long time in Passive mode

I can't even FTP to this external server in normal FTP mode, I have to use Passive Mode:
To be expected, if the external server is behind a firewall not allowing a standard ftp connection..

If this is true
instant, but then the actual upload appears to take something like three or four minutes.

and this is true
A desktop FTP client upload is also more or less instant for the same file.

then you are making the connection and transferring the file. Therefore passive/active is certainly not the problem.

I am guessing your problem is one or a combination of TCP window sizing (mismatches across the transit interfaces), or MTU/sizing or path MTU discovery enabling/disabling.

Here's why:


A desktop ftp client probably has dynamic auto window sizing, negotiating the transfer sizing all along, or some have FTP window sizing negotiation with the server at that level . . So it's possible that your client ftp app on a desktop system is going to transfer the file successfully based on its own capabilities, starting out with small window sizing, and figuring out optimizing the transfer before doing it.

And that may not be the case at all on your Linux server on the transfers to the external Windoze server.

The logon is instant, the CWD is instant, the rename of the old file with ".bak" is instant
All of which make perfect sense since these operations are invoked with small commands, all of which fit within one packet and are executed either locally or remotely without further data payload to transfer

Now if you're root on the box you'll have the luxury of changing the following settings on Linux (2.4 kernel?), which could be affecting things, if they have been set away from the usually very good defaults. Look in /proc/sys/net/ipv4 at the tcp tuning stuff

These are all normally set to "1" 
/proc/sys/net/ipv4/tcp_timestamp
/proc/sys/net/ipv4/tcp_window_scaling
/proc/sys/net/ipv4/tcp_sack

[download]

And this is usually set to "0" 
/proc/sys/net/ipv4/ip_no_pmtu_disc

[download]

And then you have the send and receive socket buffer sizes 
/proc/sys/net/core/rmem_default      (default receive window)
/proc/sys/net/core/rmem_max          (maximum receive window)
/proc/sys/net/core/wmem_default        (default send window)
/proc/sys/net/core/wmem_max            (maximum send window)

/proc/sys/net/ipv4/tcp_rmem    (memory reserved for TCP rcv buffers)
/proc/sys/net/ipv4/tcp_wmem    (memory reserved for TCP snd buffers)

[download]

Here's the values on my default system: 
/proc/sys/net/ipv4 # cat tcp_wmem
                       4096    16384   131072

/proc/sys/net/ipv4 # cat tcp_rmem
                       4096    87380   174760

[download]

I don't see that Net::FTP or Net::Config contain ways to handle transfer size negotiation. But at least you could check if your base system has been drastically altered... or there may be some wierd issue between the firewall and Windows box interface negotiating..

If it's not any of this stuff, please post the answer if you find one.. this kind of stuff also happens a lot with scp, sftp across certain "broadband routers" and crap 802.11b hardware...

If all else fails, it's perfectly permissible to blame the Microsoft system on the other endpoint:)

This and also this one are great resources..

Replies are listed 'Best First'.
Re^2: Net::FTP takes a very long time in Passive mode
by Cody Pendant (Prior) on Oct 16, 2004 at 05:12 UTC
    Thanks so much for all that detail. I'm sorry to have to tell you that the machine doing the transfer to the Windows machine isn't a LINUX box, it's actually Mac OS X. There are actually three webservers involved.

    The process is:

    1. Script on OS X webserver uploads and parses files.
    2. Script on OS X webserver FTPs to internal LINUX box (all goes well)
    3. Script on OS X webserver FTPs to NT/Firewall box (long delay)

    So I need to find out how to tweak those values on the Mac server.


    ($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss')
    =~y~b-v~a-z~s; print
[reply]
      Ah.

      Well on OSX that should be done with sysctl, which should be fairly analogous to the old FreeBSD sysctl. There are less values than Linux but you can set these at least: 
      net.inet.tcp.sendspace: 32768                                         
+                           
net.inet.tcp.recvspace: 57344                                         
+                           
net.inet.tcp.path_mtu_discovery: 1

      [download]

      You should be able to use (as root) this form 
      sysctl -w net.inet.tcp.recvspace=40960

      [download]

      or in /etc/sysctl.conf set the value with this form 
      net.inet.tcp.recvspace=40960

      [download]
      so any values are maintained across reboots.

      So I suspect the thing to look for is your default send window being too large. If that was the case the delay could be caused by excessive packet reassembly problems, with more hops to the NT probably making it worse. I would also look on the NT box at the firewall logs when you connect .. PS I don't know how close those commands are but it should get you close..
[reply]
[d/l]
[select]
        I said I'd get back to you on Monday ... it's Wednesday now but what the hell... OK I set the values, just as you said, with 
        sudo sysctl -w net.inet.tcp.recvspace=40960

        [download]

        Etc., but it doesn't seem to have made any difference.

        The upload takes just over two minutes for a one-line text file.

        But I turned on Debugging in Net::FTP -- should have done that before, shouldn't I? And found something interesting. 

        Net::FTP=GLOB(0xfc84)>>> ALLO 5
Net::FTP=GLOB(0xfc84): Timeout at ftptest.pl line 17
Net::FTP=GLOB(0xfc84)>>> PASV
Net::FTP=GLOB(0xfc84)<<< 227 Entering passive mode (202,6,74,98,243,97
+)
Net::FTP=GLOB(0xfc84)>>> STOR adsf.txt
Net::FTP=GLOB(0xfc84)<<< 125 Data connection already open; Transfer st
+arting.
Net::FTP=GLOB(0xfc84)<<< 226 Transfer complete.

        [download]

        It seems it's trying to do a non-passive transfer, (line 17 is the "put()" command), timing out after two minutes, then issuing the PASV command and doing it instantly ... what's that all about?

        Update: I issued the PASV command myself before the "put()" command, but it made no difference. It still timed out and then did its own PASV command.

        Update-Update: I realised a couple of seconds ago that I could just set the timeout to some low value and it would instantly time out then move on to the PASV thing. But I'd still like to know what's going on...


        ($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss')
        =~y~b-v~a-z~s; print
[reply]
[d/l]
[select]
        Thanks a lot, again. I'll report back on Monday.

        ($_='kkvvttuubbooppuuiiffssqqffssmmiibbddllffss')
        =~y~b-v~a-z~s; print
[reply]

In Section Seekers of Perl Wisdom