in reply to Parallel::ForkManager is time consuming...takes too long

If you do something non-trivial, fork can be a win. When you do something as trivial as a single print, the overhead makes fork a lose.
  • Comment on Re: Parallel::ForkManager is time consuming...takes too long

Replies are listed 'Best First'.
Re^2: Parallel::ForkManager is time consuming...takes too long
by esolkc (Acolyte) on Aug 12, 2011 at 18:45 UTC
    Thank you for you reply, Well, I simplified. For example, replace print($string,"\n"); with ($ouput1, $output2, $output3 ) = &some_subroutine($string); whereas some_subroutine is definded as follows: 
    sub some_subroutine 
{
        my ($string) = @_;
    my ($output1, $output2, $output3);

    if (  $string eq $some_other_string_a )
    {
           $output1 = 1350;
           $output2 = 9234;
           $output3 = 8889;
    }
    if ( $string eq $some_other_string_b )
    {
           $output1 = 1345;
           $output2 = 2234;
           $output3 = 3689;
    }
    if ( $string eq $some_other_string_c )
    {
           $output1 = 2256;
           $output2 = 3370;
           $output3 = 1340;
    }
    @retval = ($output1,$output2,$output3);
}

    [download]
    so the code looks like 
    #!/usr/bin/perl
use Parallel::ForkManager;

@all_strings=('110110101000101111', 
              '110010101001011101', 
              '110110111001001110',
              '110011101001011100',
              '111010101001011100');

my $pm = new Parallel::ForkManager(4);
foreach $string (@all_strings)
{
  my $pid = $pm->start and next;
  ($ouput1, $output2, $output3 ) = some_subroutine($string);
  $pm->finish;
}
$pm->wait_all_children;

    [download]
    Or do you have an simple example that really works?
[reply]
[d/l]
[select]
      Still too trivial. Do something that takes some noticeable amount of time. The simplest example would be, e.g., sleep for a few seconds. I often use Parallel::ForkManager for things that individually take a few minutes, it's not much good for things that take a tiny fraction of a second.
[reply]
        Is a double loop over 1,000,000 and 20,000 not big enough for fork. I need the process to be finished in less than 30 sec., while using all cores. The sleep() example is difficult to compare to the issue I outlined. I could not find in the documentation how the forking is implemented.
[reply]

      $string == $some_other_string_a

      btw, this is buggy. One needs to use eq to compare strings.

[reply]
[d/l]
[select]
      Running a hundred Perl ops and doing a number of system calls including forking a process in order to avoid running a dozen Perl ops in series is indeed a losing proposition.
[reply]
        At least I have the syntax right. So but when should I use forking? All examples that I can find somewhere in the Net are as simple as the example I have provide. I have several loops and subroutines which can be distributed on different CPU-cores.
[reply]

In Section Seekers of Perl Wisdom