Re^2: No Performance gain with Parallel::ForkManager

Hi,

probably something like that:

#!/usr/bin/perl
use strict;
use warnings;
use strict;
use 5.010;

use local::lib './lib';
use File::Find::Rule;
use Parallel::ForkManager;
use utf8;

my $dir = './';

my @mp3s = File::Find::Rule->file()
    ->name( '*.mp3' )
    ->in($dir);

say "INFO: Number of mp3 files found " . scalar @mp3s;

my $num_of_cores = 2;

my $pm = Parallel::ForkManager->new($num_of_cores);
for my $proc_id (0..$num_of_cores - 1) {
    my $pid = $pm->start and next;

    for(my $i = $proc_id; $i < scalar @mp3s; $i += $num_of_cores) {
        my $file = $mp3s[$i];
        say "Proc: $proc_id: Working on '$file'";
    }
    $pm->finish;
}
$pm->wait_all_children;
say "END";
[download]

I'm pretty sure you see the building blocks.

Regards
McA

Comment on Re^2: No Performance gain with Parallel::ForkManager Download Code

Replies are listed 'Best First'.
Re^3: No Performance gain with Parallel::ForkManager by RichardK (Parson) on Feb 23, 2014 at 17:59 UTC
There's no need to be quite that verbose as Parallel::ForkManager handles the number of processes for you. So you can write that as :- `... my $pm = Parallel::ForkManager->new($num_of_cores); for my mp3 (@mp3s) { $pm->start and next; do_work($mp3); $pm->finish; } $pm->wait_all_children;` [download]	[reply] [d/l]
Re^4: No Performance gain with Parallel::ForkManager by McA (Priest) on Feb 23, 2014 at 18:29 UTC
Hi Richard, am I wrong or does your solution fork a child per file found? I wanted to give walto a snippet where I show how to spawn as many subprocesses as cores are available and the subprocesses working on a subqueue. The verbose code tries to show that the subprocesses iterate over the initially created array in a way that they can "share" () this array without doing work twice. Have I overseen something? UPDATE: It was an answer to the question "I can not find a simple way to split the loop into 2 (my no of cores) independent subprocesses...". Best regards McA () It's a copy in the subprocess.	[reply]
Re^3: No Performance gain with Parallel::ForkManager by walto (Pilgrim) on Feb 23, 2014 at 19:06 UTC
Thanks for your example! Adding an extra loop for the no of cores brought an speed improvement of the parallel execution of 100%. Time parallel execution real 0m0.312s, user 0m0.304s, sys 0m0.052s Time serial exection real 0m0.324s, user 0m0.272s, sys 0m0.031s That's approx 4% overall gain.	[reply]