in reply to Return all the data from child to parent with Parallel::Forkmanager

fork (and in turn, Parallel::ForkManager) doesn't return data to the parent process and has no provisions to do so.

The easiest approach would be to have each client write out the results to a file, and then read each file in turn in the parent. The run_on_finish callback allows you to conveniently know when to collect more data in the parent:

use JSON::XS 'encode_json', 'decode_json'; $pm->run_on_finish(sub { my ($pid, $exit_code, $filename) = @_; print "** $ident was just written"; open my $fh, '<', $filename or die "Couldn't read child results +from '$filename': $!"; binmode $fh; local $/; my $results = <$fh>; # Convert results back to the data structure: $results = json_decode( $results ); # now, merge those results with the other results you already ha +ve }); ... for my $task (@tasks) { my $filename = "some filename based on $task"; $pm->start( $filename ) and next; ... do the work open my $fh, '>', $filename or die "Couldn't write output file '$filename': $!"; binmode $fh; print $fh json_encode(\%results); };

Replies are listed 'Best First'.
Re^2: Return all the data from child to parent with Parallel::Forkmanager
by afoken (Chancellor) on Aug 18, 2017 at 12:51 UTC
    fork (and in turn, Parallel::ForkManager) doesn't return data to the parent process and has no provisions to do so.

    Nearly right. fork doesn't, but recent versions of Parallel::ForkManager (starting at 0.7.6, released 2010-Aug-15) do allow returning a single data structure, that must be a reference to a string, a hash, or an array. See "RETRIEVING DATASTRUCTURES from child processes" in the documentation of Parallel::ForkManager. What happens there is quite similar to your JSON approach: Parallel::ForkManager serialises the data in the child process using Storable, writes it to disk, and reads it back in the parent, all nearly transparent to the user.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)