metaperl has asked for the wisdom of the Perl Monks concerning the following question:

I'm wondering what the simplest way is to have a set of forked processes all have access to a data element, preferably a data element which has lexical not package scope.

As you can see below, even with package scope, there is no data in @out, even though it is a package variable: 


#!/usr/bin/perl -w

use strict; use warnings;

use Parallel::ForkManager;
use LWP::Simple;
my $pm=new Parallel::ForkManager(10);

my @in = 1..5 ;
our @out;

for my $link (@in) {
  $pm->start and next;

  sleep rand 50 ;

  die if $link == 2 ;

  push @out, $link ;

  $pm->finish;

};

$pm->wait_all_children;

warn "@out"

[download]

Replies are listed 'Best First'.
Re: Easy shared data for forked processes?
by shmem (Chancellor) on May 20, 2009 at 13:56 UTC
    there is no data in @out

    See the synopsis: 

    foreach $data (@all_data) {
    # Forks and returns the pid for the child:
    my $pid = $pm->start and next;

    ... do some work with $data in the child process ...

    $pm->finish; # Terminates the child process
}

    [download]

    Note $data in the child process. Changes to data in the child doesn't affect the data in the parent. See the various threads about "setting environment variables from system() in perl" or such.

    If you use fork, you have to return the data from the child to the parent via some IPC mechanism. But I guess you'd be better off if you use threads, pre-allocate an array with references and have each thread modify its slot in that array.

[reply]
[d/l]
Re: Easy shared data for forked processes?
by Anonymous Monk on May 20, 2009 at 13:41 UTC
[reply]
Re: Easy shared data for forked processes?
by morgon (Priest) on May 20, 2009 at 15:05 UTC
    Have a look at IPC::Shareable if you are running on a sytem with shared memory.
[reply]
Re: Easy shared data for forked processes?
by metaperl (Curate) on May 20, 2009 at 19:09 UTC
    Yuval Kogman came up with a KiokuDB solution: 
    
#!/usr/bin/perl -w

use strict;
use warnings;

use Time::HiRes qw(sleep);
use Test::More 'no_plan';

{

    package Simple;
    use Moose;

    has 'nums' => (
        isa     => 'ArrayRef[Int]',
        is      => 'rw',
        default => sub { [] },
    );

}

use KiokuDB;
use Parallel::ForkManager;

my @in = 1 .. 50;
our @out;

my $pm = new Parallel::ForkManager(5);

my $dir = KiokuDB->connect( "dbi:SQLite:dbname=kiokudb_tutorial.db", c
+reate => 1 );

my ( $id ) = $dir->txn_do( scope => 1, body => sub { $dir->insert( Sim
+ple->new ) } );

warn "id: $id";

for my $link (@in) {
    $pm->start and next;

    retry: {
        eval {
            $dir->txn_do(
                scope => 1,
                body  => sub {
                    my $obj = $dir->lookup($id);

                    push @{ $obj->nums }, $link;

                    $dir->update($obj);
                }
            );
        };

        if ( $@ ) {
            warn "$$ failed to commit, sleeping for random interval an
+d retrying";
            my $i = rand(0.1);
            sleep $i;
            redo retry;
        } else {
            warn "$$ successfully comitted";
        }
    };

    $pm->finish;

}

$pm->wait_all_children;

$dir->txn_do( scope => 1, body => sub {
    my $obj = $dir->lookup($id);
    
    is( scalar(@{ $obj->nums }), scalar(@in), "update count is correct
+" );

    warn $obj->dump;
});

    [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom