in reply to Bandwidth limiting for file downloads: What can Perl do?

I've looked at such limiters previously, and the most things I find are rate limiters (but they are more central to my interests too). There is HTTP::Tiny::Bandwidth, which claims to do bandwidth limiting, but I haven't used it. Maybe this would work for you to slow down downloads.

For hacking this into LWP::UserAgent, simply doing a usleep 0.5 or whatever in the content_cb callback would lower the bandwidth used, but it will still be quite bursty. I'm not aware of any convenient solution there, or for the async toolkits, so if you find anything relevant, a post about this is welcome!

Replies are listed 'Best First'.
Re^2: Bandwidth limiting for file downloads: What can Perl do?
by etj (Priest) on Apr 29, 2022 at 13:32 UTC
    Conceivably, Mojo::UserAgent might have features that could help with this.
[reply]
Re^2: Bandwidth limiting for file downloads: What can Perl do?
by Polyglot (Chaplain) on Apr 30, 2022 at 12:18 UTC
    I appreciate the lead. If you or someone else could give me a clue as to where to add this delay and how to make the callback, it would be much appreciated. I'm unfamiliar with the methods used by LWP::UserAgent, and when I tried using the special ':content_cb'     => \&callback mechanism, with "callback" being a subroutine that used usleep, all I got for my effort was a downloaded file named /content_cb ...hardly the expected behavior. I'm obviously clueless, even after reading the documentation. Given a hint as to how to start this, I'd like to see if anything useful could be done to rate-limit the connection. I spent a few hours playing with it already, but couldn't get past even the first step.

    Update: I found a question asked some years ago on SO that posted a code snippet which invoked the ':content_cb' callback, and I tried it. It changes absolutely nothing except that the downloaded file is misnamed (becomes "/content_cb", instead of the .html file that it should be). No change whatsoever in speed, regardless of the number put into "usleep". Adding a print line resulted in nothing as well--no output to the screen, so I don't think the subroutine is actually running, and it seems LWP::UserAgent is conflating the ':content_cb' token for the ':content_file' one.

    Blessings,

    ~Polyglot~

[reply]
[d/l]
[select]

      The following program downloads a file very slowly. Maybe this gets you started.

      #!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
use 5.020;
use feature 'signatures';
no warnings 'experimental::signatures';

my $mech = WWW::Mechanize->new();

my $large_url = 'http://ftp.acc.umu.se/mirror/wikimedia.org/dumps/dewi
+ki/20220420/dewiki-20220420-abstract.xml.gz';

$| = 1;

my $read_size = 0;
$mech->get(
    $large_url,
    ':read_size_hint' => 4096,
    ':content_cb' => sub {
        $read_size += length( $_[0] );
        my $len = length($_[0]);
        print "\r$len - $read_size bytes";
        # discard the content
        sleep 1;
    },
);

say 'done';

      [download]
[reply]
[d/l]
        So why might that work with WWW::Mechanize and not with LWP::UserAgent which is supposed to accept the same callbacks? Is this a bug in the latter? With the Mechanize, it did reduce the bandwidth considerably, but at the end of 10+ minutes I had no file. How does one keep the file, too?

        Blessings,

        ~Polyglot~

[reply]

In Section Seekers of Perl Wisdom