in reply to Re: OT: Parallel::ForkManager and suitable load (Ubuntu)
in thread OT: Parallel::ForkManager and suitable load (Ubuntu)

I'm in the process of setting up a new machine that has six cores, so perhaps that box will be better at handling this multi-kid version.

Interestingly, the original server had stability problems, so I reduced the number of kids from 50 down to 20, and this only increased the run time from 15 minutes to 18 minutes. The load was about half the original.

Alex / talexb / Toronto

Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

  • Comment on Re^2: OT: Parallel::ForkManager and suitable load (Ubuntu)

Replies are listed 'Best First'.
Re^3: OT: Parallel::ForkManager and suitable load (Ubuntu)
by marioroy (Prior) on Mar 12, 2024 at 16:48 UTC
    > Interestingly, the original server had stability problems, ...

    Greetings, talexb. That can be from many kids initiating a connection simultaneously in a tiny window. The following is a P::FM like code. The interesting bit is MCE::Hobo->yield, helpful for staggering connection instantiations. MCE::Hobo->yield defaults to 0.008 seconds on UNIX. Call yield prior to making a remote connection.

    use v5.10.1; use MCE::Hobo; MCE::Hobo->init( max_workers => 50, posix_exit => 1, on_finish => sub { my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @ +_; print "child $pid completed: $ident => ", $resp->[0], "\n"; } ); foreach my $data ( 1..2000 ) { MCE::Hobo->create( $data, sub { MCE::Hobo->yield(0.008); # sleep 1; # simulate connection instantiation [ $data * 2 ]; }); } MCE::Hobo->wait_all;

    Without the "sleep 1", the demonstration completes in ~ 0.8 seconds (without yield) and ~ 16.0 seconds (with yield). Otherwise, ~ 42 seconds simulating work. No matter how many workers, the serial-delay capability will not allow more than 125 connection instantiations per second with MCE::Hobo->yield(); default 0.008 seconds.

    For use-cases like this, serialized delay/yield mitigates many workers initiating a connection concurrently in a tiny window, improving reliability.

      MCE::Hobo uses MCE::Shared for IPC. You may prefer MCE::Child to not involve a shared-manager process. MCE::Child uses MCE::Channel for IPC. MCE::Channel came long after making MCE::Hobo and thought to make something similar that works with Perl v5.8.1 and above.

      use v5.10.1; use MCE::Child; MCE::Child->init( max_workers => 50, posix_exit => 1, on_finish => sub { my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @ +_; print "child $pid completed: $ident => ", $resp->[0], "\n"; } ); foreach my $data ( 1..2000 ) { MCE::Child->create( $data, sub { MCE::Child->yield(0.008); # sleep 1; # simulate connection instantiation [ $data * 2 ]; }); } MCE::Child->wait_all;

      Writing MCE::Child was helpful in improving MCE::Hobo and vice versa. Yeah, there are two similar modules. MCE::Child is mostly compatible with MCE::Hobo. MCE::Hobo is more compatible with threads, but requires Perl v5.10.1 minimally.