controlling child processes

ministry has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monks,

I'm developing a script that will retrieve log files via a raptor (firewall) specific application called 'remotelogfile'.

Goal:
Fork a predetermined amount of child processes, each using the remotelogfile utility for simultaneous collection. The next child process must wait to be spawned until the number of child processes goes below the predetermined amount of children.

Caveat:
I can't do a collection of all firewalls listed in my array at the same time, i.e. - the wait.

Questions:
One - how can I make the script wait until all child processes have finished, before exiting?
Two - should I be decrementing the $counter every time a loop completes
Three - is there a better way to go about doing this?

Here's a perlmonks friendly version of what I have so far...

my($num_procs)='3';
my($counter)='0';
foreach my $fw qw(fw1 fw2 fw3 fw4 fw5){
  $counter++;
  if($counter <= $num_procs){
    &do_stuff($fw);
  }
  else{
    wait;
    &do_stuff($fw);
  }
}

sub do_stuff{
  my($fw)=shift;
  local $SIG{CHLD}='IGNORE';
  defined(my $pid=fork) || die "Can not fork!";
    exec "remotelogfile $fw logfile > /var/$fw.log" if(!$pid);
}
[download]

Any help would be most appreciated.

cheers!
-Ev

Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.

Comment on controlling child processes Download Code

Replies are listed 'Best First'.
Re: controlling child processes by ikegami (Patriarch) on Apr 04, 2006 at 22:10 UTC
Three - is there a better way to go about doing this? Yes. You are duplicating the functonality of Parallel::ForkManager. All you need is the following: `use strict; use warnings; use Parallel::ForkManager (); use constant $MAX_PROCESSES => 3; { my $pm = Parallel::ForkManager->new($MAX_PROCESSES); foreach my $fw (qw(fw1 fw2 fw3 fw4 fw5)) { my $pid = $pm->start and next; exec "remotelogfile $fw logfile > /var/$fw.log" or die("Unable to launch \"remotelogfile $fw\": $!\n"); # Will never reach this, but that's ok since it only calls exit. $pm->finish; # Terminates the child process } }` [download] or just `use Parallel::ForkManager (); my $pm = Parallel::ForkManager->new(3); foreach my $fw (qw(fw1 fw2 fw3 fw4 fw5)) { $pm->start and next; exec "remotelogfile $fw logfile > /var/$fw.log"; die("Unable to launch \"remotelogfile $fw\": $!\n"); }` [download]	[reply] [d/l] [select]
Re: controlling child processes by bluto (Curate) on Apr 04, 2006 at 23:03 UTC
It is easy to get these problems wrong, so I second using Parallel::ForkManager. In your solution you need to keep looping not only if there is more work to spawn, but also if there are children to reap. Update: That's confusing to read -- it just means you'll need to reap 3 children at the end. I like to use a hash to maintain the children. Here's an untested solution, probably with bugs ... `use strict; use warnings; $SIG{CHLD} = 'IGNORE'; my $max_procs = 3; # don't use strings when you want numbers my @work = qw(fw1 fw2 fw3 fw4 fw5); my %kids; while (@work or keys %kids) { if (@work and $max_procs > keys %kids) { my $fw = shift @work; my $pid = fork; die "Can not fork!" if ! defined $pid; if ($pid) { $kids{$pid} = $fw; } else { exec "remotelogfile $fw logfile > /var/$fw.log"; die; } } else { my $pid = waitpid(-1, 0); delete $kids{$pid}; } }` [download]	[reply] [d/l]
Re: controlling child processes by johngg (Canon) on Apr 04, 2006 at 23:02 UTC
I have a script that does tape backups. It forks a child process for each tape device on the host so that multiple tape devices are written concurrently rather than consecutively. This is the skeleton of the process, which uses pipes so that the children can log messages that the parent can read later. use IO::Pipe; ... foreach $tapeDevice (@tapeDevices) { $pipeFH = IO::Pipe->new(); ... if($pid = fork()) { # This is the parent. $pipeFH->reader(); push @pipes, $pipeFH; ... } elsif(defined $pid) { # This is the child. $pipeFH->writer(); # Do lots of stuff writing to tape ... $pipeFH->print("some message or other"); ... } else { # Fork failed for some reason. die "fork: $!\n"; } } # We have now forked a child for each tape device so # wait for our children to finish. Print exit status. while((my $returnPID = wait()) != -1) { print "Child $returnPID returned status ", $? >> 8, "\n"; } # All children have returned, get messages. foreach $pipe (@pipes) { while(defined($_ = $pipe->getline())) { # Do something ... } } [download] That is the bare bones of the mechanism. I have not attempted to reconcile `use strict;` and `my` or `our` here as it is just an illustration. I hope this is useful. Cheers, JohnGG	[reply] [d/l] [select]


Clear questions and runnable code get the best and fastest answer
	PerlMonks