Re^2: Fork vs pThreads

I have an unknown number of streams.

Every 5 seconds I have a new one that is stored in a table. I then execute a select with a limit of 50, the number of rows returned(nrRowsReturned) goes into the fork pool.

my $pm = Parallel::ForkManager->new($nrRowsReturned);

while(there are rows to fetch){
  $pm->start and next;
  processStream($stream);
  $pm->finish;
}
$pm->wait_all_children();
[download]

Comment on Re^2: Fork vs pThreads Download Code

Replies are listed 'Best First'.
Re^3: Fork vs pThreads by Corion (Patriarch) on Oct 21, 2013 at 14:07 UTC
Why are you running as many children in parallel as you have rows? For 1000 rows, you will launch 1000 children in parallel. The idea of Parallel::Forkmanager is to use the optimal number of parallel children, which is usually roughly the number of CPUs (or cores) of your machine, and not the number of tasks to be processed. Other people have already recommended a smaller number of concurrent processes, like 4 or 8. Maybe you should try that suggestion.	[reply]
Re^4: Fork vs pThreads by ThelmaJay (Novice) on Oct 21, 2013 at 14:15 UTC
Thank you :) I didn't realized that. It makes sense in term of cpu usage. But I still have the same problem right? If one of the 4 takes longer time to end, the mother still needs to wait for that to finish to be able to send another 4 right?	[reply]
Re^5: Fork vs pThreads by Corion (Patriarch) on Oct 21, 2013 at 15:02 UTC
As far as I understand Parallel::ForkManager, it limits your program to maximum 4 children, but if one child ends, Parallel::ForkManager will launch the next child immediately.	[reply]