What OS? How much RAM on the machine? Do you have swap configured?

I'm going to assume a variant on Unix because one child dies and the rest continue. At a wild guess, you've got 2 GB of RAM and no swap. In which case you are running out of RAM. Adding more RAM or configuring swap should stop the problem. Having swap will slow things down a lot.

Another possible fix is to split your data into a larger number of smaller pieces, then use something like Parallel::ForkManager to process it with a fixed number of children at any time. That will give you the parallelism you're looking for while controlling how much memory you need at any one time. Make the size of the target pieces be fixed. That way as your dataset continues to grow, your memory needs will stay fixed.

In reply to Re: child process dies soon after fork() call by tilly
in thread child process dies soon after fork() call by haidut

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.