Im trying to use Parallel::ForkManager to split off some slow processes in hopes of increasing overall performance. The 'main' task is one that should lend itself well to parallel activity: examining 'client data' in a db by way of 10-15 queries each of which reports some value. There are ~100 'clients' in the db. The current application takes each client in sequence, runs the series of queries and saves the results in a table.

The vast majority of the processing time is spent waiting for the DB server. The server is running MySQL on an 8 way box with 8G of RAM. The box running the application(s) is a dual core with 1G of RAM.

Here's what I did and what confused me:
I used the sample(s) from Parallel::ForkManager to rewrite my app so that instead of looping through the list of available 'clients' it gets them all in an array and calls 
 foreach $row ( @{$ref} ) {
     my $pid = $pm->start(@$row->[0]) and next;
     my $out = `./alert_main.pl -h $row->[0] -p $passId`;
     print $out;
     print "\n";
     $pm->finish($row);
 }

[download]

So it spins off the number of processes that ForkManager is called with, each runs, exits and the next one starts. I see it doing its work via the callbacks that ForkManager provides. All well and good. Here's where Im confused:
I run the application with 'max_procs' set to 1 and I get a total run time thats equivalent to my old 'one at a time' method. When I change 'max_procs' to 8 it takes more than 4 times longer to complete. I watch the CPU activity on the DB box and MySQL is (apparently) using all 8 procs. The total load runs at around 600%. The box that the app is running on never goes above 5%.

What am I doing wrong?

In reply to fork-ing : my lack of understanding by ethrbunny

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.