Joost++.

It will be less of an hassle to use external programs optimized for downloading like wget or curl instead of trying to use LWP. LWP is a general purpose library to interact with websites and it sometimes croaks when downloading large files. Also, LWP does not have support for downloading entire directories and resuming aborted downloads. All that logic has to be implemented in your code. With wget/curl - you get all these "built-in" and your perl code is a simple wrapper.

offtopic: In college, we had a thin pipe connection and we could not download research papers without timeouts, etc. I used to create a file with the list of URLs to fetch, run wget-f <filename> in a detached screen session and it would download the entire file overnight. I think you need to do something similar here.

Mahesh

In reply to Re^2: Automatic Download of Multiple Files with Perl by smahesh
in thread Automatic Download of Multiple Files with Perl by monkfan

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.