I glossed over the initial processing of the file because I didn't think it was relevant to the problem at hand for two reasons, but I'll expand.

The input file is processed by open-software bio-informatics tools called "bowtie2" and "samtools". They are invoked by use of a system() call, and they write temporary files. All of that seems to work fine.

After those programs are finished, my program (which I inheritied, btw) does its forking and further processing.

So here's the funny thing. This program is running successfully on a MacPro with 48Gb of RAM and 24 CPU cores. On my linux box, with 128Gb of RAM and 40 CPU cores, the program fails as I've described. I'm assuming, because of the way that it's failing (some of the child processes die immediately with an error code of 11 (EAGAIN), that I'm running out of some critial resource and my fork is failing. What I can't figure out, and the answer that I'm seeking, is which resource? Is it memory? Swap space? Open file descriptors? Something else? How do I figure this out, other than trial-and-error raising of each limit in turn, which is what I've been trying to do.

So am I entirely confident my algorithms are correct? Given that they're running just fine on a smaller machine, I'm reasonably confident that they are, and that my problem is somehow related to the number of forks I'm running (one per logical CPU core).

In reply to Re^2: Trouble with fork() by phonybone_monk
in thread Trouble with fork() by phonybone_monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.