As tye pointed out, using threads or forks is a way to keep things moving. While some processes hang, others can continue. As the slow processes time out, they are cleared away to make room for new processes.

And you definitely want to put a throttle on this, especially if you are talking about potentially hundreds of processes!

You might want to consider merlynīs parallelizing engine which is mentioned in this informative perlmonks discussion, as well as his forking parallel link checker column which he mentioned in response to some similar code submitted by kschwab earlier.

Matt

P.S. I was going to dust off my so-called "Hatchery" module from my non-released bag of tricks, but considering Randallīs columns I think it would be pretty redundant. sigh.It is amusing to note that what he called "hungry puppies" I tended to call "screaming kids", although I did refer to them as a litter. Well, perhaps once I finish my trip Iīll polish it up and post a comparitive analysis if there seems to be enough differences to warrant it.

In reply to Re: Dealing with timeouts using the Shell module by mojotoad
in thread Dealing with timeouts using the Shell module by thunders

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.