adamquestion has asked for the wisdom of the Perl Monks concerning the following question:

I am using the following code to run a series of processes at the same time. I have an 8 processer server, thus I limit the number of runs at one moment to 5, then start processes. The following code is fine, but it waits for all the 5 processes to finish in order to start the next 5. It would have been better if I start with 5 and replace the one which is finished immediately with a new one, thus have 5 processes running at same time always. Any ideas? Thanks Adam 
DATELOOP:
while ( ($currentdate <= $enddate) && ($item < 5) )
{
    $item++;
    pipe *{$item},RETHAND;
    unless ($pid = fork()) {    
        print "$command\n";
        system($command);
        print RETHAND "Completed $currentdate\n";
        exit();
     }
     $currentdate = nextdate($currentdate);
}

foreach $item2 (@waitlist) {     
         $response = <$item2>;  # received from $item
            print "$response\n";
            $item--;
}

if  ($currentdate <= $enddate) {goto DATELOOP;}

[download]

Replies are listed 'Best First'.
Re: forks for running multiple codes at same time
by Corion (Patriarch) on Feb 09, 2009 at 07:28 UTC
[reply]
Re: forks for running multiple codes at same time
by Anonymous Monk on Feb 09, 2009 at 02:25 UTC
[reply]
      or Proc::Queue: 
      use Proc::Queue size => 5, # => 5 parallel processes max.
                qw(system_back);

use POSIX ":sys_wait_h";
use Errno qw(EINTR);

my %pid2date;

sub reap_children {
  my $wait = shift;
  my $pid = waitpid(-1, ($wait ? 0 : WNOHANG));
  if ($pid > 0) {
    print "process for date $pid2date{$pid} (pid $pid) exited with cod
+e ". ($? >> 8) . "\n";
    return 1;
  }
  return ($! == EINTR);
}

while ($currentdate <= $enddate) {
  reap_children(0);
  print "$command\n";
  my $pid = system_back($command);
  $pid2date{$pid} = $currentdate;
  $currentdate = nextdate($currentdate);
}

1 while reap_children(1);

      [download]
[reply]
[d/l]
Re: forks for running multiple codes at same time
by weismat (Friar) on Feb 09, 2009 at 05:45 UTC
    From my point of view you may consider to change to threads instead of fork.
    Every fork call has a significant overhead.
    Send all tasks on a queue and every thread reads from the queue and calls the wanted function.
[reply]

      This is untrue.

      This is only true for Win32, where the fork() call doesn't even really exists (it's poorly faked using threads), on every other platform fork() costs almost the same, and provides many valuable additional benefits (like better memory protection, better management etc etc)

      Not to mention, he's spawning only 5 (count'em - five) processes, what would be cummulative gain on that, even on win32?

[reply]
        I agree that there is no difference if you would create as many threads as forked processes.
        The difference is that you need to call fork once for every planned task - whereas in a thread model you can share data and thus you would start threads only 5 times depending on the number of workers as the workers can get the data from a queue instead of a function parameter.
[reply]

Back to Seekers of Perl Wisdom