Receiving data from many children

xiper has asked for the wisdom of the Perl Monks concerning the following question:

I have a parent that spawns a bunch of children, up to 10 at any one time, to do some 'mostly-idle' processing. My question is, how to i get the results back from each child? Previously i've used the exit status of the child, which is fine for simple return values, but now i have a complex-data-structure to be returned (although can easily be encoded into a string by Data::Dumper if necessary). I've experimented with pipes but they only seem to work with 1 parent - 1 child relationship. IPC::Shareable looks to do what i want, but i wanted to get input from others to see if there was a simpler solution.

My code so far, if anyone's interested:

my( %pid );

foreach my $num ( "01" .. "25" )
{
  # if max processes, wait for one to finish
  if( keys( %pid ) >= 10 ) { &reap() }
  
  if( my $pid = fork() )  # parent
  {
    $pid{$pid} = $num;
  }
  else                    # child
  {
    &child( $num );
    exit 0;
  }
}

# reap remaining processes
while( %pid ) { &reap() }

# <collect data>

sub reap
{
  my( $pid ) = wait();
  
  if( $pid == -1  ) { die( "no more kids to reap!\n" ) }
  if(! $pid{$pid} ) { die( "why did I see $pid?\n"   ) }
  
  delete( $pid{$pid} );
}

sub child
{
  my( $num ) = $_[0];
  print "child $num spawned\n";
  # <send data>
}
[download]

Thanks all!

- ><iper

my JAPH:
  print"Just another Perl hacker\n";
  #    ^ look, no space! pretty tricky huh?
[download]

Comment on Receiving data from many children Select or Download Code

Replies are listed 'Best First'.
Re: Receiving data from many children by Zaxo (Archbishop) on Aug 20, 2003 at 03:09 UTC
You can open a pipe before you start spawning the children. Each child closes the read side of the pipe, and prints data to the write side. The parent reads from the pipe, which has become shared among the kids by duplication of the handle. I published a similar example here at Many-to-One pipe. There, however, I had one child collecting the data from the others. You can adapt that to your requirements. After Compline, Zaxo	[reply]
Re: Re: Receiving data from many children by xiper (Friar) on Aug 20, 2003 at 04:21 UTC
Hmm, i tried experimenting with pipes before i posted but couldn't get them to work... Just tried again, copying your example, and tada, works! Finally figured out that in my first attempt i had the reader/writer mixed up in the initial pipe... slaps self Future readers, the end result: Read more... (1443 Bytes)	[reply] [d/l] [select]
•Re: Receiving data from many children by merlyn (Sage) on Aug 20, 2003 at 03:16 UTC
Either use POE or look at the code in my column on getting your kids to do the work. -- Randal L. Schwartz, Perl hacker Be sure to read my standard disclaimer if this is a reply.	[reply]
Re: Receiving data from many children by Flame (Deacon) on Aug 20, 2003 at 16:06 UTC
You could also try the Threads implementation if you're running 5.8.0. There, you can simply return values and the parent can get them via the join command. True, Threads are still kinda experemental... Hey, it's your call... My code doesn't have bugs, it just develops random features. Flame ~ Lead Programmer: GMS (DOWN) \| GMS (DOWN)	[reply]
Re: Receiving data from many children by NetWallah (Canon) on Aug 21, 2003 at 00:02 UTC
Here is a working sample of multiple threads. Tested on Activeperl 5.8 on Win32. `use strict; use threads; my @MyThread; my $threadcount = 15; for (0..($threadcount-1)){ $MyThread[$_]=threads->new(\&s,$_,10+$_); }; if ($ARGV[0] eq 'a'){ foreach (@MyThread){ print qq(Return check via thread array from ) . $_->join() . qq(\n +); } }else{ foreach (threads->list()){ print qq(Return check via threads list from ) . $_->join() . qq(\n +); } } ####################### sub s{ my $a=shift; sleep($threadcount-$a); print qq(In sub s param=$a\n); return shift() }` [download]	[reply] [d/l]


laziness, impatience, and hubris
	PerlMonks