Maybe i'm missing something, but in neither of the 2 replies posted so far do i see anything that seems to address wilstephens's specific question about limiting to amount of parrallellisssmmmm to a pecific number (his example was 10).

I've never used LWP::Parallel, but after a quick glance at the docs (the concept intrigues me) I notice this method for LWP::Parallel::RobotUA ... 

  $pua->max_req  ( 2);  # max parallel requests per server

[download]
but that's not quite what we're looking for.

I can't for the life of me see anything obvious here. I would guess you could do this by only registering 10 URLs at a time, getting them in parrallel, then registering another 10, etc. But this is wastefull of cycles. Some URLs will finish faster then others, but you're wiating for all 10 before proceeding to #11.
(I would guess there is a better way)

UPDATE: LWP::Parallel::UserAgent also seems to have a max_req method, as well as a max_hosts method -- so if you know that all of your URLs wer for different hosts, OR were all for the same host, you could use one of them to get your result, but i still don't see anyway to optimally fetch no more then 10 URLs a time without any pre previous knowledge of your URLs. method

In reply to Re: Using LWP::Parallel by hossman
in thread Using LWP::Parallel by wilstephens

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.