I actually had a very similar sort of problem a while back, where the files I was trying to download were quite large, but I never managed to resolve it using LWP.

However, while this may not be what you want to do, I managed to resolve my own problem by using curl instead, and setting a limit-rate on the downloads. i.e. I used my same Perl script for everything, except for the downloads I simply issued a system call from my script to kick off curl with the options I provided.

You sound like somebody who may already know about curl, in which case you'll already know there are a gazillion options for it. My downloads were mostly FTP, but it works equally well for http, or whatever.

The option you'd want to use with curl is "--limit-rate".

For example, one of my commands looks similar to this: 

$cmd="curl -u $uname:$pwd -O --limit-rate 50K ftp://$host$dir$file"

[download]
... where 50K is the limit-rate which I've set, but you can set it to whatever you like (e.g. 10K, 35K, etc).

I'm sure there are also probably curl options for getting past the certificate too.

In reply to Re: LWP Browser->Get Challenge by DanielSpaniel
in thread LWP Browser->Get Challenge by rkellerjr

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.