If you know wget's syntax better than LWP, and your target system has wget, use system() and wget. There is nothing wrong with using Perl for shell scripting. LWP has higher CPU/MEM overhead than the C wget. The core LWP (libwww-perl) doesn't support async requests. Backgrounding command line downloaders is a easy way to implement async internet requests in Perl. I personally use http://aria2.sourceforge.net/ as my command line downloader. Sometimes I run aria2 in daemon mode and control it through XML::RPC::Fast having it downloading dozens of files at the same time to disk, then I read the files synchronously into the Perl from disk for processing, its almost instantaneous because of OS write caching.

In reply to Re: Using LWP instead of wget? by bulk88
in thread Using LWP instead of wget? by kingram

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.