in reply to Fork vs pThreads

I think your basic logic error is here:
my $pm = Parallel::ForkManager->new(50); # 50 forks my $i = 0; while(i<50){ # start 50 forks $pm->start and next; processStream($stream); $i++; $pm->finish; # do the exit in the child process } $pm->wait_all_children(); # and then do nothing
I'd amend your code to this (adjusting the fork-amount to something sane such as 4):
# max 4 processes simultaneously my $pm = Parallel::ForkManager->new(4); while( .. there are rows to fetch ..){ $pm->start and next; processStream($stream); $pm->finish; # do the exit in the child process } $pm->wait_all_children();
The amount of rows to fetch could be 50 or whatever number that you want. Parallel::ForkManager will take care of the queueing and will keep four forks alive as long as there are jobs in the queue.

Replies are listed 'Best First'.
Re^2: Fork vs pThreads
by ThelmaJay (Novice) on Oct 21, 2013 at 16:18 UTC
    Exactly :). You guys are the best!! :) I already see improvements :). Thank you all for your replies! :)