I just wanted to make a comment real quick. Make sure that you are not falling to fallatious logic in believing that somehow calling forth the gods of "explicit parallelism" you will be guaranteed to incurr a speedup. Remember, in situations like this, where you are *pulling* on the dataflow, and more importantly, your data set rests on a node to which you do not have a guaranteed transfer rate, it is possible that parallelizing your GET's will not increase your actual throughput or minimize your wall-clock time for the entire transaction.

Remember your Von Neumann bottleneck, it is doubtful that what is slowing down your task is the overhead of processing the data, it is much more likely that the bottleneck exists in the actual data pipe (in other words, not processing BUT bandwidth!) And attempting to stuff 10k/sec of data down a 1k/sec pipe won't make the pipe bigger... it may actually end up slowing down your overall wall-clock time due to TCP collisions and other assorted baddies. I'm a big fan of parallelization throughout... just make sure it makes *sense* in your particular configuration.

In reply to Re: Downloading URL's in Parallel with Perl by eduardo
in thread Downloading URL's in Parallel with Perl by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.