Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^5: Useful number of childs revisited

by BrowserUk (Patriarch)
on May 08, 2015 at 18:47 UTC ( [id://1126144]=note: print w/replies, xml ) Need Help??


in reply to Re^4: Useful number of childs revisited
in thread Useful number of childs revisited [SOLVED]

i don't understand this.

See the "Blocking calls" section of the P::FM docs.

By default; the parent process polls to see if one fo the existing kids has completed; and sleeps between polls. And by default, sleeps's smallest granularity is 1 second. Thus, it does not notice when a child process has finished for upto 1 second after it has.

For processes as in (my version of) your example where the processing takes 0.142s to complete; that means each process that gets waited for takes 85% longer in elapsed time before its replacement gets started.

If the module used the 4-arg select instead of sleep to control it polling loop; it would be able to used a much smaller timeout, and notice when child processes finished in a more timely manner.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Replies are listed 'Best First'.
Re^6: Useful number of childs revisited
by karlgoethebier (Abbot) on May 08, 2015 at 19:53 UTC

    Now i feel like i'm walking on very thin ice.

    You suggested to set $pm->set_waitpid_blocking_sleep(0);  # true blocking calls enabled.

    After setting this in my/your script it seems as it behaves like as expected, right?

    You suggest to switch sleep for select( '','','', $timeout ).

    I assume that you refer to line 782 of the sources:

    sub _waitpid_blocking { my $self = shift; # pseudo-blocking if( my $sleep_period = $self->{waitpid_blocking_sleep} ) { while() { my $pid = $self->_waitpid_non_blocking; return $pid if $pid; sleep $sleep_period; # should be line 782 } } return waitpid -1, 0; }

    The author names it $sleep_period instead of $timeout.

    Do you mean that sleep $sleep_period should be replaced with select( '','','', $sleep_period )?

    I'm sorry if it looks like i got a true block. Please correct me if i'm wrong.

    Update: Do'h! It's too late tonight!

    Edit: No reason to complain.

    Thank you very much for further advice and best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

      Do you mean that sleep $sleep_period should be replaced with select( '','','', $sleep_period )?

      Yes. Because: sleep uses whole seconds; whereas select "The timeout, if specified, is in seconds, which may be fractional.... You can effect a sleep of 250 milliseconds this way:select( undef, undef, undef, 0.25 );".

      That would allow you to avoid setting $pm->set_waitpid_blocking_sleep(0);; by instead setting set_waitpid_blocking_sleep( $seconds ); to a much lower -- but non-zero! -- value than the standard one whole second (say:0.01) which would also achieve the goal of avoiding the pregnant pause between one child process ending and its replacement being started.

      The reason why this latter method is preferable to setting the timeout (sleep period) to 0; is documented in the "BLOCKING CALLS" section of the P::FM docs. It basically comes down to the fact that when the timeout is set to 0; instead of using wait to wait for particular child processes; it uses waitpid to wait for any child process. In most circumstances that will be fine because the only child processes will be those that were started by, and thus being managed by, P::FM.

      But, if the parent process is also starting other child processes, then waitpid can also be woken by one of those other children that shouldn't be under P::FMs control; thus a) the code that started this 'other' child won't be able to obtain its return code; b) P::FM will start another child thinking that one of its child processes has ended, when it hasn't.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

        Thank you very much for this in-depth explanation.

        Update:

        I applied your advice:

        karls-mac-mini:monks karl$ ./buks_patch.pl 0.01 processes: 4 fork: 13.8262140750885 karls-mac-mini:monks karl$ ./buks_patch.pl 0.25 processes: 4 fork: 13.9020979404449 karls-mac-mini:monks karl$ ./buks_patch.pl 0.5 processes: 4 fork: 15.0743520259857 karls-mac-mini:monks karl$ ./buks_patch.pl 1.0 processes: 4 fork: 20.0701560974121

        It seems like it works as expected, very nice!

        For completeness, the modified code:

        #!/usr/bin/env perl use strict; use warnings; use Math::BigInt; use Parallel::ForkManager; use Time::HiRes qw ( time ); use feature qw(say); my $sleep_period = shift; my $processes = qx(sysctl -n hw.ncpu); chomp $processes; my $pm = Parallel::ForkManager->new($processes); $pm->set_waitpid_blocking_sleep($sleep_period); say qq(processes: $processes); my $start = time; for my $number ( 1 .. 50 ) { $pm->start($number) and next; my $factorial = Math::BigInt->bfac(2000) for 1 .. 10; $pm->finish(0); } $pm->wait_all_children; say qq(fork: ), time - $start;

        ForkManager.pm:

        sub _waitpid_blocking { my $self = shift; # pseudo-blocking if( my $sleep_period = $self->{waitpid_blocking_sleep} ) { while() { my $pid = $self->_waitpid_non_blocking; return $pid if $pid; # sleep $sleep_period; select( '','','', $sleep_period ); } } return waitpid -1, 0; }

        My best regards, Karl

        «The Crux of the Biscuit is the Apostrophe»

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1126144]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (3)
As of 2024-03-29 02:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found