mikeym has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I have a set of resources that I would like to process asynchronously, unfortunately I don't have the authority to install one of the perl modules that would make this a sinch, like POE. So I'm falling back on the oldschool Fork and process method. However I'm finding choosing the best IPC method difficult and I was wondering if someone here could help. This is the kind of structure I'm thinking about:

#!/bin/perl use strict; # setup shared resource between children and parent my @resource = qw(thing1 thing2 another_thing some more things which m +ust be more than the pool of processes); my @todo = (0..$#resource); # indexes of shared resources to process my $parent = $$; my @children; # spawn pool of child processes for (1..10) { # setup 2 way communication (pipe?) ... my $pid; unless ($pid = fork()) { # child process # stop if signaled local $SIG{HUP} = sub { $end = 1 }; # work until signaled to die while(1) { # read instruction from parent my $message = ... # if finished let the child die if ($end or $message eq "END") { # send message that child is exiting ... exit(); # if we have recieved an index to a shared res +ource to process do so } elsif ($message =~ /^[0-9]+/) { if ($message >= @resource) { # send message that child is exiting ... die "Resource unavailable\n"; } # process shared resource process($resource[$message]); } else { # send message that child is exiting ... die "Caught unexpected communication\n"; } } } push @children, $pid; } # keep count of the number of living children $kids = @children; print "Forked $kids children\n"; # send messages to children to begin processing ... for(@todo[0..$#children]); while (@children > 0) { # read messages from children into $message and $child ... # receive index of shared resource if complete if ($message =~ /^[0-9]+$/) { # remove from todo list @todo = grep { $_ != $message } @todo; last unless (@todo); # stop if no more to do # message $child to do next @todo item ... } elsif($message eq "END") { # message from child of exit # remove child from pool @children = grep { $_ != $child } @children; } } # make sure the children have stopped kill 'HUP', @children; print "Complete\n";

Replies are listed 'Best First'.
Re: Processing a set of resources with a pool of forked children
by zentara (Cardinal) on Feb 01, 2014 at 11:47 UTC
    You probably would be better off figuring out how to install Parallel::ForkManager.

    Here is the bare bones way to do it. You didn't say exactly what your problem was in the code.

    #!/usr/bin/perl use warnings; use strict; my %pids; my @to_be_processed = (1..20); my $cmd = 'echo'; foreach my $p (@to_be_processed){ $pids{$p}{'pid'} = open($pids{$p}{'fh'}, "$cmd $p 2>&1 |"); } foreach my $key(keys %pids){ my $fh = $pids{$key}{'fh'}; while (<$fh>){ print $_; #stuff data here into a return hash } } foreach my $key(keys %pids){ waitpid($pids{$key}, 1); } print "done\n";

    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
Re: Processing a set of resources with a pool of forked children
by zentara (Cardinal) on Feb 01, 2014 at 12:18 UTC
Re: Processing a set of resources with a pool of forked children
by shmem (Chancellor) on Feb 01, 2014 at 22:02 UTC

    I'd go with IO::Select and IO::Handle (maybe IO::Pipe as well), these modules are core. The following may serve as inspiration.

    use IO::Handle; use IO::Select; my %pids; # kid job description keyed on pid my $kids_fh; # pipe filehandles foreach kid keyed on fileno my $rset = IO::Select->new; local $| = 1; local $SIG{CHLD} = sub { my $waitedpid = wait; print "process $waitedpid ($pids{$waitedpid}) finished" if $waited +pid > 0; }; # spawn childeren for my $cmd( @cmds ) { ; my ($rh,$wh); for ($rh,$wh) { $_ = IO::Handle->new; $_->autoflush; } pipe($rh,$wh); if ((my $pid = fork) == 0) { # child code } else { # parent $rset->add($rh); $pids{$pid} = $cmd; $kids_fh{fileno($rh)} = $pid; } } # now interact with kids while(1) { last unless $rset->handles; foreach my $rh (sort { fileno $a <=> fileno $b } $rset->can_read) +{ my $response = do { local $/; <$rh> }; ... ... $rset->remove($rh); } }
    perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
      foreach my $rh (sort { fileno $a <=> fileno $b } $rset->can_read) {

      Why sort the ready handles by fileno?


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Good catch, thank you. Well, er... first come, first serve. Children always have an eye on each others privileges. ;-)

        Seriously, it's an artifact from cut&paste. In the code I took that from, processing had preferably to be in the same order as the spawned children. Not a technical requirement.

        perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
Re: Processing a set of resources with a pool of forked children
by Pickwick (Beadle) on Feb 01, 2014 at 20:53 UTC
    So I'm falling back on the oldschool Fork and process method.
    Why don't you have a look at threads and Queues? This way you can share data easily and you can start as many workes as you like.
Re: Processing a set of resources with a pool of forked children
by Bloodnok (Vicar) on Feb 03, 2014 at 13:55 UTC
    You may not have authority to install to system locations, but you should be able to install to an alternative location by using the PREFIX macro when invoking perl e.g. perl Makefile.PL PREFIX=$HOME/lib
    A user level that continues to overstate my experience :-))