in reply to No Performance gain with Parallel::ForkManager

I'm a bit late but perhaps this might be still of interest:

Using splice for building the subqueues:

my @mp3s = ( 'a' .. 'z' ); my $cores = 4; my @queue; push @queue, [ splice @mp3s, 0, $cores ] while @mp3s;

Or an iterator with List::MoreUtils:

use List::MoreUtils qw (natatime); my @mp3s = ( 'a' .. 'z' ); my $cores = 4; my @queue; my $iterator = natatime $cores, @mp3s; while ( my @buff = $iterator->() ) { push @queue, \@buff; }

This yields:

$VAR1 = [ [ 'a', 'b', 'c', 'd' ], [ 'e', 'f', 'g', 'h' ], [ 'i', 'j', 'k', 'l' ], [ 'm', 'n', 'o', 'p' ], [ 'q', 'r', 's', 't' ], [ 'u', 'v', 'w', 'x' ], [ 'y', 'z' ] ];

Then:

use Parallel::ForkManager; my $cores = 4; my $pm = Parallel::ForkManager->new($cores); foreach my $child (@queue) { $pm->start and next; process($child); $pm->finish; } $pm->wait_all_children; sub process { my $mp3s = shift; foreach my $mp3 (@$mp3s) { # do the stuff } }

Regards, Karl

«The Crux of the Biscuit is the Apostrophe»

Replies are listed 'Best First'.
Re^2: No Performance gain with Parallel::ForkManager
by walto (Pilgrim) on Feb 24, 2014 at 16:55 UTC
    Hi Karl, On Linux you can get the number of cores with
    my $no_of_cores = system ('/usr/bin/nproc');
    I also tested your approach to split the load onto each core. It was slower than McA suggegestion and the serial processing of files but faster than the original parallel processing.

    real 0m0.531s, user 0m0.449s, sys 0m0.208s

    But thanks for sharing your wisdom. I learnt a lot about making use of a multi core cpu and splitting workloads.
      " ...your approach to split the load...It was slower..."

      Seems like i lost once more ;-)

      "There seems to be something wrong with our bloody ships today." (David_Beatty,_1st_Earl_Beatty)

      Best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

        It was faster than...
        Always look on the bright side ;)