in reply to Using Net::FTP::Throttle

I suspect that the setting of the maximum burst rate to 10k (10240) here (in new()): 

    my $mbps = $arg{MegabitsPerSecond} || croak "No MegabitsPerSecond 
+passed";
    my $bps = $mbps * 1024 * 1024 / 8;

    my $bucket = new Algorithm::TokenBucket $bps, 10240;

[download]

Combined with the minimum accounting period of 1/100th of a second here (in get()):

        sleep 0.01 until $bucket->conform($len);
        $bucket->count($len);

[download]

Means that the maximum throughput possible is 10kbytes per 0.001 of a second = 1000KBytes or (roughly) 10Mbps.

Throw in the out-of-band overheads (writing to the local file), and that probably explains why it has a fixed upper limit regardless of your requested limit.

Two (3) possible solutions:

  1. Increase the max-burst value.

    Try doubling it and see what through put it gives you. Adjust accordingly.

  2. Decrease the accounting period.

    Try halving it. Adjust ...

  3. A mix of the two.

Also, 4096 byte blocksize seems very small for transferring large files across a network? I try 64k and see what affect that has.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Replies are listed 'Best First'.
Re^2: Using Net::FTP::Throttle
by joetesta (Novice) on Aug 02, 2012 at 17:57 UTC

    Thanks BrowserUK!

    With the max-burst doubled, I was seeing better throughput. So I doubled it again and it looks like now I'm seeing around 50mbps

    Regarding the block size, I thought the same, however it seemed that anything over 4K was not helping, and I only got better throughput by decreasing it to 1024 and finally 512. Perhaps now that I've increased max-burst, a larger block size wouldn't hurt?

    at least this is working now; I wonder if Net::FTP::Throttle should be updated to increase that max-burst? How does stuff like that get done, should I be prompting someone or going at it myself?

    kind regards, Joe
[reply]
      How does stuff like that get done, should I be prompting someone or going at it myself?

      I'd start by sending the author an email pointing to this thread at perlmonks and go from there.

      With regard to blocksize. I just did some not-very-representative testing using the loopback address on my machine and with 4k blocks the maximum throughput I can get is 140MB/s:

      C:\test>sock-rated.pl -O=4 -I=4 1000
svr: request for 1048576000 
1048576000 took 6.999 seconds. [142.89 MB/s]

      [download]

      By using 100k blocks, I can get that up to over 300MB/s:

      C:\test>sock-rated.pl -O=100 -I=100 10000
svr: request for 10485760000 
10485760000 took 33.032 seconds. [302.74 MB/s]

      [download]

      Not a real-world test, but still 4k seems like a pifflingly small amount.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom