You are actually making your program slower by forking! You are doing so little in the children that deserializing the results from the child is slower than actually generating them in the first place! You need to do a lot more work per child.

You had the right idea about including more levels of looping in the child. 

use Algorithm::Loops      qw( NestedLoops );
use Parallel::ForkManager qw( );

my $num_jobs_log6 = 4;   # 6*6*6*6 jobs.
my $num_workers = 64;    # 20.25 jobs/worker (on average).

my $pm = Parallel::ForkManager->new($num_workers);

my @param_arr;
$pm->run_on_finish(sub{
   my ($pid, $exit_code, $ident, $exit_signal, undef, $data) = @_;
   die("Child killed by signal $exit_signal\n") if $exit_signal;
   die("Child return error $exit_code\n") if $exit_code;

   push @param_arr, @$data;
});

my @ps = map { $_ * 0.2 } 0..5;
my $outer_iter = NestedLoops([ (\@ps) * $num_jobs_log6 ]);
while (my @p_outer = $outer_iter->()) {
   my $p_outer = join("\t", @p_outer);
   $pm->start() and next;

   my @child_output;
   my $inner_iter = NestedLoops([
      (\@ps) * ( 11 - $num_jobs_log6 ),
      [ 1 ], 
      [ 1 ],
   ]);
   while (my @p_inner = $inner_iter->()) {
      push @child_output, $p_outer . join("\t", @p_inner) . "\n";
   }

   $pm_finish(0, \@output);
}

$pm->wait_all_children();

[download]

I'm still not convinced that this will be faster than doing it all in one process. You do more work in the child, but you also increase the amount of time needed to deserialize the result of the child.

I think you are optimizing the wrong part of your code! If this part of the code is really slowing you down, just pre-generate the values in a file and load that file in the array. 

my @param_arr;
{
   open(my $fh, '<', $qfn)
      or die("Can't open \"$qfn\": $!\n");

   @param_arr = <$fh>;
}

[download]

In reply to Re: Parallelization of multiple nested loops by ikegami
in thread Parallelization of multiple nested loops by biosub

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.