I saw roughly the same speed differences as everyone else, both using the external URL in your script and using the internal file that I started with. After running a profiler on LWP, it looks like most of the time is spent appending the new chunk it just downloaded to the existing buffer (HTTP::Message, add_content method, line 142 in version 5.814):
$self->{_content} .= $$chunkref;
add_content is called in a loop by the collect method of LWP::Protocol. I was able to reduce the download time for a 12MB file from 36 seconds to 5 seconds by changing the collect method to use an array for the buffer and then stick everything together at the end, instead of calling add_content repeatedly:
my @total_content; if (!defined($arg) || !$response->is_success) { # scalar while ($content = &$collector, length $$content) { if ($parser) { $parser->parse($$content) or undef($parser); } LWP::Debug::debug("read " . length($$content) . " bytes"); push(@total_content, $$content); $content_size += length($$content); $ua->progress(($length ? ($content_size / $length) : "tick"), +$response); if (defined($max_size) && $content_size > $max_size) { LWP::Debug::debug("Aborting because size limit exceeded"); $response->push_header("Client-Aborted", "max_size"); last; } } $response->add_content(join('', @total_content));
Anybody know any reason that would be a problem?

In reply to Re^2: LWP slow downloads on windows (50 times faster?) by robobunny
in thread LWP slow downloads on windows by robobunny

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.