in reply to Fork vs pThreads

How many units of work do you have overall? Are there only 50 units of work, or are there more than 50?

Your current setup only processes the first 50 units of work and does not process any further items.

If the code you show is not the code you are really running, you'll need to show some more relevant code.

Replies are listed 'Best First'.
Re^2: Fork vs pThreads
by ThelmaJay (Novice) on Oct 21, 2013 at 14:02 UTC

    I have an unknown number of streams.

    Every 5 seconds I have a new one that is stored in a table. I then execute a select with a limit of 50, the number of rows returned(nrRowsReturned) goes into the fork pool.

    my $pm = Parallel::ForkManager->new($nrRowsReturned);

while(there are rows to fetch){
  $pm->start and next;
  processStream($stream);
  $pm->finish;
}
$pm->wait_all_children();

    [download]
[reply]
[d/l]

      Why are you running as many children in parallel as you have rows? For 1000 rows, you will launch 1000 children in parallel.

      The idea of Parallel::Forkmanager is to use the optimal number of parallel children, which is usually roughly the number of CPUs (or cores) of your machine, and not the number of tasks to be processed.

      Other people have already recommended a smaller number of concurrent processes, like 4 or 8. Maybe you should try that suggestion.

[reply]
        Thank you :) I didn't realized that. It makes sense in term of cpu usage. But I still have the same problem right? If one of the 4 takes longer time to end, the mother still needs to wait for that to finish to be able to send another 4 right?
[reply]

In Section Seekers of Perl Wisdom