in reply to Parallelization of multiple nested loops
You are actually making your program slower by forking! You are doing so little in the children that deserializing the results from the child is slower than actually generating them in the first place! You need to do a lot more work per child.
You had the right idea about including more levels of looping in the child.
use Algorithm::Loops qw( NestedLoops ); use Parallel::ForkManager qw( ); my $num_jobs_log6 = 4; # 6*6*6*6 jobs. my $num_workers = 64; # 20.25 jobs/worker (on average). my $pm = Parallel::ForkManager->new($num_workers); my @param_arr; $pm->run_on_finish(sub{ my ($pid, $exit_code, $ident, $exit_signal, undef, $data) = @_; die("Child killed by signal $exit_signal\n") if $exit_signal; die("Child return error $exit_code\n") if $exit_code; push @param_arr, @$data; }); my @ps = map { $_ * 0.2 } 0..5; my $outer_iter = NestedLoops([ (\@ps) * $num_jobs_log6 ]); while (my @p_outer = $outer_iter->()) { my $p_outer = join("\t", @p_outer); $pm->start() and next; my @child_output; my $inner_iter = NestedLoops([ (\@ps) * ( 11 - $num_jobs_log6 ), [ 1 ], [ 1 ], ]); while (my @p_inner = $inner_iter->()) { push @child_output, $p_outer . join("\t", @p_inner) . "\n"; } $pm_finish(0, \@output); } $pm->wait_all_children();
I'm still not convinced that this will be faster than doing it all in one process. You do more work in the child, but you also increase the amount of time needed to deserialize the result of the child.
I think you are optimizing the wrong part of your code! If this part of the code is really slowing you down, just pre-generate the values in a file and load that file in the array.
my @param_arr; { open(my $fh, '<', $qfn) or die("Can't open \"$qfn\": $!\n"); @param_arr = <$fh>; }
|
|---|