Re^2: LWP slow downloads on windows (50 times faster?)

I saw roughly the same speed differences as everyone else, both using the external URL in your script and using the internal file that I started with. After running a profiler on LWP, it looks like most of the time is spent appending the new chunk it just downloaded to the existing buffer (HTTP::Message, add_content method, line 142 in version 5.814):

    $self->{_content} .= $$chunkref;
[download]

add_content is called in a loop by the collect method of LWP::Protocol. I was able to reduce the download time for a 12MB file from 36 seconds to 5 seconds by changing the collect method to use an array for the buffer and then stick everything together at the end, instead of calling add_content repeatedly:

    my @total_content;

    if (!defined($arg) || !$response->is_success) {
    # scalar
    while ($content = &$collector, length $$content) {
        if ($parser) {
        $parser->parse($$content) or undef($parser);
        }
        LWP::Debug::debug("read " . length($$content) . " bytes");
        push(@total_content, $$content);
        $content_size += length($$content);
        $ua->progress(($length ? ($content_size / $length) : "tick"), 
+$response);
        if (defined($max_size) && $content_size > $max_size) {
        LWP::Debug::debug("Aborting because size limit exceeded");
        $response->push_header("Client-Aborted", "max_size");
        last;
        }
    }
    $response->add_content(join('', @total_content));
[download]

Anybody know any reason that would be a problem?

Comment on Re^2: LWP slow downloads on windows (50 times faster?) Select or Download Code

Replies are listed 'Best First'.
Re^3: LWP slow downloads on windows (50 times faster?) by BrowserUk (Patriarch) on Dec 04, 2008 at 17:52 UTC
I can't see any problem with that. You might save a little more time by avoiding copying the strings twice: `push(@total_content, $content); ... $response->add_content(join('', map $$_, @total_content));` [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^3: LWP slow downloads on windows (50 times faster?) by mr_mischief (Monsignor) on Dec 05, 2008 at 22:39 UTC
Your changes indeed speed things up on my testing system so long as they don't cause additional swapping. I see only one issue. It does seem to use more memory than the stock LWP::Protocol. Yours hit more heavily into my swap file for a large file (703MB on a system with only 512 MB of RAM) than the stock version. The map in BrowserUk's version put my poor antiquated testing server right out of physical RAM and swap completely. My conclusion is that I really do need more RAM in this machine. It's something to consider if you're under similar constraints or if you're submitting a patch to the module's maintainers, though. As always, paging to swap can erase any gains you'd get otherwise and then cause even much slower execution. Sometimes optimizing for speed at the expense of memory fails, and sometimes optimizing for memory use is actually a performance win.	[reply]
Re^3: LWP slow downloads on windows (50 times faster?) by kszyska (Initiate) on Apr 05, 2009 at 23:50 UTC
This helped greatly speeding up zip downloads when I added this fix to the collect method in LWP::Protocal under perl 5.825. Before the fix it took ~3.5h to download 28 zip files and after adding this LWP::Protocal fix it took only ~1.5h :-) However, now I have perl 5.8.9 (with latest libwww-perl 5.825 2009-02-16) installed and the zip downloads are back to taking ~3.5 hours. I diffed the Protocal.pm files but they are redical different (between libwww-perl 5.825 2009 and Protocol.pm,v 1.41 2003/10/23 19:11:32) and I don't see any hooks for a similar quick fix. Can anyone suggest a fix for speeding up downloads using libwww-perl (5.825 2009-02-16)...?	[reply]