I would be looking for ways to make the child processes use less memory. (Are input files being slurped when they could be processed one record at a time? Is each child making unnecessary copies of its input data, e.g. by reading a whole file into a scalar then splitting into an array? Are there complex data structures where simpler storage would do? Would it make sense to use additional disk-based resources instead of in-memory data structures, e.g. dbm files or other database(-like) storage?)

Failing that, I'd be checking whether it's really necessary to have four children running at once. What does that quantity get you that you don't get with two consecutive jobs with two children per job?

If there is just one factor that makes the difference between "it works" and "it fails", and that one factor is the size of the input files, and it turns out that these files are just always getting bigger, you've got a scaling problem, which is a sort of design problem. Anything that doesn't solve the design problem is just going to be a stop-gap, temporary fix with a limited life-span.

Solving the design problem is a matter of figuring out how to complete the task within a finite amount of ram, such that the process runs with a stable and consistent footprint no matter what size the input data may be.

In reply to Re: child process dies soon after fork() call by graff
in thread child process dies soon after fork() call by haidut

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.