If speed is the key, but process-spawning is too slow, threads are what you need. As you said, Perl Threads are experimental. To use many C threading, i believe you would end up finding it easier to write the program in C/C++. You might want to look into using fork() to spawn 50 processes, and have each get two files, or some such configuration.
I would think of fork() as the most 'perl' way of doing it in the end, but i have never used LWP::Parallel, and would assume it is based on IO Multiplexing (good section in Network Programming with Perl), which is complex, and eventually the disk IO will cause slowness with large amounts of data, but i would benchmark to find out if it is any quicker than the fork version.
speling champ of tha claz uf 1997
-- MZSanford

In reply to Re: Downloading URL's in Parallel with Perl by MZSanford
in thread Downloading URL's in Parallel with Perl by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.