Re^3: OT: Parallel::ForkManager and suitable load (Ubuntu)

> Interestingly, the original server had stability problems, ...

Greetings, talexb. That can be from many kids initiating a connection simultaneously in a tiny window. The following is a P::FM like code. The interesting bit is MCE::Hobo->yield, helpful for staggering connection instantiations. MCE::Hobo->yield defaults to 0.008 seconds on UNIX. Call yield prior to making a remote connection.

use v5.10.1;
use MCE::Hobo;
 
MCE::Hobo->init(
    max_workers => 50,
    posix_exit  => 1,
    on_finish   => sub {
        my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @
+_;
        print "child $pid completed: $ident => ", $resp->[0], "\n";
    }
);
 
foreach my $data ( 1..2000 ) {
    MCE::Hobo->create( $data, sub {
        MCE::Hobo->yield(0.008);
        # sleep 1;  # simulate connection instantiation
        [ $data * 2 ];
    });
}
 
MCE::Hobo->wait_all;
[download]

Without the "sleep 1", the demonstration completes in ~ 0.8 seconds (without yield) and ~ 16.0 seconds (with yield). Otherwise, ~ 42 seconds simulating work. No matter how many workers, the serial-delay capability will not allow more than 125 connection instantiations per second with MCE::Hobo->yield(); default 0.008 seconds.

For use-cases like this, serialized delay/yield mitigates many workers initiating a connection concurrently in a tiny window, improving reliability.

Comment on Re^3: OT: Parallel::ForkManager and suitable load (Ubuntu) Download Code

Replies are listed 'Best First'.
Re^4: OT: Parallel::ForkManager and suitable load (Ubuntu) by marioroy (Prior) on Mar 12, 2024 at 17:17 UTC
MCE::Hobo uses MCE::Shared for IPC. You may prefer MCE::Child to not involve a shared-manager process. MCE::Child uses MCE::Channel for IPC. MCE::Channel came long after making MCE::Hobo and thought to make something similar that works with Perl v5.8.1 and above. `use v5.10.1; use MCE::Child; MCE::Child->init( max_workers => 50, posix_exit => 1, on_finish => sub { my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @ +_; print "child $pid completed: $ident => ", $resp->[0], "\n"; } ); foreach my $data ( 1..2000 ) { MCE::Child->create( $data, sub { MCE::Child->yield(0.008); # sleep 1; # simulate connection instantiation [ $data * 2 ]; }); } MCE::Child->wait_all;` [download] Writing MCE::Child was helpful in improving MCE::Hobo and vice versa. Yeah, there are two similar modules. MCE::Child is mostly compatible with MCE::Hobo. MCE::Hobo is more compatible with threads, but requires Perl v5.10.1 minimally.	[reply] [d/l]

Replies are listed 'Best First'.

Re^4: OT: Parallel::ForkManager and suitable load (Ubuntu)
by marioroy (Prior) on Mar 12, 2024 at 17:17 UTC

MCE::Hobo uses MCE::Shared for IPC. You may prefer MCE::Child to not involve a shared-manager process. MCE::Child uses MCE::Channel for IPC. MCE::Channel came long after making MCE::Hobo and thought to make something similar that works with Perl v5.8.1 and above.

use v5.10.1;
use MCE::Child;
 
MCE::Child->init(
    max_workers => 50,
    posix_exit  => 1,
    on_finish   => sub {
        my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @
+_;
        print "child $pid completed: $ident => ", $resp->[0], "\n";
    }
);
 
foreach my $data ( 1..2000 ) {
    MCE::Child->create( $data, sub {
        MCE::Child->yield(0.008);
        # sleep 1;  # simulate connection instantiation
        [ $data * 2 ];
    });
}
 
MCE::Child->wait_all;
[download]

Writing MCE::Child was helpful in improving MCE::Hobo and vice versa. Yeah, there are two similar modules. MCE::Child is mostly compatible with MCE::Hobo. MCE::Hobo is more compatible with threads, but requires Perl v5.10.1 minimally.

[reply]
[d/l]