Hello, perlmonks,

I am seeking advice on diagnosing a problem related to a Perl program (which I inherited) that makes use of fork(). The basic problem seems to be that some of the forked sub-processes immediately exit without doing any work. When those processes are reaped (with wait()), $? is set to 11, which I believe is the error code for "resource temporarily unavailable, try again". I am trying to determine which resource is unavailable. The environment is Linux/Centos 7

The basic structure of this program is as follows:

• - Initialization
• - Read in a large file (~10Gb)
• - Do some processing of the input file using other open-source software (bowtie and samtools), via calls to system(). This results in the writing of some intermediate files.
• - Process the intermediate files in parallel. For each processor core unit (as reported by Sys::Info::Device::CPU, or by reading /proc/cpuinfo), my program calls fork(). But instead of then calling exec(), the child continues processing on its own.
• - The parent process, after spawning all the children, enters a loop around a call to wait(). If wait() returns -1, the parent prints a message that includes the value of $?, and then exits (via Carp::confess).

Here's a pseudo code outline (note: this won't actually re-create my problem, of course):

# Initialization
# Read input file, calls to system('bowtie2', ...) and system('samtool
+s', ...)
use strict;
use warnings;
use Carp qw(confess);
use Sys::Info::Device::CPU; 


use Sys::Info::Constants qw( :device_cpu );
my $info = Sys::Info->new;
my $cpu  = $info->device( CPU => {} );
my $nthreads = $cpu->count;
print "$nthreads threads\n";

while ($nthreads--) {
    my $pid = fork;
    confess ("Can't fork: $!") unless defined $pid;
    if (! $pid) {
    print "child process: $$\n";      # this line never gets executed 
+by some of the children
    # Do a bunch of processing using intermediate files as input, usin
+g Bio::DB::Sam->get_features_by_location() to access intermediate fil
+es
    exit 0;
    }
}

# parent continues here:
my $pid = 0;
my $ok = 1;
while ($pid != -1) {
  $pid = wait;
  last if $pid == -1;

  my $err=$?;
  if ($err) {
    print("one of the children died: pid=$pid, err=$err\n");
    $ok = 0;
  }
}
if (! $ok) {
  confess "some child processes died.";
}
print "all done, yay\n";
[download]

In all of the processes, the line directly after the fork confess("Can't fork: $!") unless defined $fork; never gets executed. I infer that the fork itself did happen.

But some of the processes, depending on the input file, are reaped immediately, with $err=11 (EAGAIN). I surmise I am running short of some resource, but which one? I have increased the limits for the following resources to values (by editing /etc/security/limits.conf, and then rebooting) that I believe should be more than sufficient, to no avail:

• - number of open file handles
• - number of allowed processes per user
• - number of file locks allowed
• - amount of lockable memory

I've also increased swap space via swap files.

Short or restructuring the code to use exec(), which I've considered, do any wise monks have suggestions as to:

• - how to determine which missing resource is causing fork() to exit with $?=EAGAIN?
• - any other useful areas to examine?

My thanks for all answers, insights, and useful comments