Parallel:ForkManager how to pass back a list

cormanaz has asked for the wisdom of the Perl Monks concerning the following question:

I have code where a forked process is generating a list in $data and passing that to the callback like in this simplified version:

use strict;
use feature ':5.10';
use Parallel::ForkManager;

our $pm = Parallel::ForkManager->new(8);

$pm -> run_on_finish ( # called BEFORE the first call to start()
    sub {
        my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data_
+structure_reference) = @_;

        # retrieve data structure from child
        if (defined($data_structure_reference)) {  # children are not 
+forced to send anything
             my $string = ${$data_structure_reference};  # child passe
+d a string reference
             if ($string) { 
                  #how to process $string? 
             };
        }
        else {  # problems occurring during storage or retrieval will 
+throw a warning
            say qq|No message received from child process $pid!|;
        }
    }
);

LOOP:
foreach my $instance (0..100) {
    $pm->start and next LOOP; # do the fork
    my $data;
    foreach my $i (0..50) {
        push(@$data,int(rand(10));
    }
    $pm->finish(0,\$data); # exit the child process
}

$pm->wait_all_children;
[download]

The docs say you can use this method to return a data structure, but they refer to the thing returned as a string. Is $string really a string or is it an object?

Comment on Parallel:ForkManager how to pass back a list Download Code

Replies are listed 'Best First'.
Re: Parallel:ForkManager how to pass back a list by davido (Cardinal) on Apr 05, 2024 at 18:04 UTC
It is an actual data structure reference. I have this in some of my code somewhere: `$pm->run_on_finish( sub { $buckets->{$_[0]} = $_[1] for @{$_[5]}; } );` [download] So that means that `$_[5]` (`$data_structure_reference`) is shaped something like: `$data_structure_reference = [ ['a', 1], ['e', 1], ['l', 1], ['p', 2], ]` [download] Keep in mind the following information from the POD for Parallel::ForkManager: The ability for the parent to retrieve data structures is new as of version 0.7.6. Each child process may optionally send 1 data structure back to the parent. By data structure, we mean a reference to a string, hash or array. The contents of the data structure are written out to temporary files on disc using the Storable modules' store() method. The reference is then retrieved from within the code you send to the run_on_finish callback. The data structure can be any scalar perl data structure which makes sense: string, numeric value or a reference to an array, hash or object. There are 2 steps involved in retrieving data structures: 1) A reference to the data structure the child wishes to send back to the parent is provided as the second argument to the finish() call. It is up to the child to decide whether or not to send anything back to the parent. 2) The data structure reference is retrieved using the callback provided in the run_on_finish() method. Keep in mind that data structure retrieval is not the same as returning a data structure from a method call. That is not what actually occurs. The data structure referenced in a given child process is serialized and written out to a file by Storable. The file is subsequently read back into memory and a new data structure belonging to the parent process is created. Please consider the performance penalty it can imply, so try to keep the returned structure small. In the case of my code example, the data structure is being layered into a hash that has been in scope since before the forked children were created. Dave	[reply] [d/l] [select]
Re: Parallel:ForkManager how to pass back a list by marioroy (Prior) on Apr 06, 2024 at 05:41 UTC
The excitement of running parallel may cause us to forget, ah yes, simply let Perl be Perl. The P::FM documentation provides a scalar example. That is send a scalar ref, de-ref as such in the `run_on_finish` callback. Ditto for array or hash. `$pm->finish(0,\$scalar); # or $pm->finish(0,\@array); # or $pm->finish(0,\%hash);` [download] `$pm -> run_on_finish ( sub { my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data_ +ref) = @_; if (defined($data_ref)) { my $scalar = ${$data_ref}; # or my @array = @{$data_ref}; # or my %hash = %{$data_ref}; } } );` [download]	[reply] [d/l] [select]
Re: Parallel:ForkManager how to pass back a list by talexb (Chancellor) on Apr 06, 2024 at 15:27 UTC
You can pass back whatever you want. I just started using this module recently, and needed to pass back a delete list and an insert list. So my `run_on_finish` sub starts off with this .. `$pfm->run_on_finish ( sub { my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data_structure_reference) = @_; if ( defined $data_structure_reference ) { my $delete_list = $data_structure_reference->{delete_list}; my $insert_list = $data_structure_reference->{insert_list};` [download] That sets things up for when a kid finishes and puts something into the return parameter. Then, in the kid code, I do this: `if ( keys %delete_list \|\| keys %insert_list ) { $pfm->finish ( 0, { delete_list => \%delete_list, insert_list => \%insert_list } ); } else { $pfm->finish; }` [download] and it just works. Magic. Alex / talexb / Toronto Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.	[reply] [d/l] [select]