I'm in the process of setting up a new machine that has six cores, so perhaps that box will be better at handling this multi-kid version.

Interestingly, the original server had stability problems, so I reduced the number of kids from 50 down to 20, and this only increased the run time from 15 minutes to 18 minutes. The load was about half the original.

> Interestingly, the original server had stability problems, ...

Greetings, talexb. That can be from many kids initiating a connection simultaneously in a tiny window. The following is a P::FM like code. The interesting bit is MCE::Hobo->yield, helpful for staggering connection instantiations. MCE::Hobo->yield defaults to 0.008 seconds on UNIX. Call yield prior to making a remote connection.

use v5.10.1;
use MCE::Hobo;
 
MCE::Hobo->init(
    max_workers => 50,
    posix_exit  => 1,
    on_finish   => sub {
        my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @
+_;
        print "child $pid completed: $ident => ", $resp->[0], "\n";
    }
);
 
foreach my $data ( 1..2000 ) {
    MCE::Hobo->create( $data, sub {
        MCE::Hobo->yield(0.008);
        # sleep 1;  # simulate connection instantiation
        [ $data * 2 ];
    });
}
 
MCE::Hobo->wait_all;
[download]

Without the "sleep 1", the demonstration completes in ~ 0.8 seconds (without yield) and ~ 16.0 seconds (with yield). Otherwise, ~ 42 seconds simulating work. No matter how many workers, the serial-delay capability will not allow more than 125 connection instantiations per second with MCE::Hobo->yield(); default 0.008 seconds.

For use-cases like this, serialized delay/yield mitigates many workers initiating a connection concurrently in a tiny window, improving reliability.

MCE::Hobo uses MCE::Shared for IPC. You may prefer MCE::Child to not involve a shared-manager process. MCE::Child uses MCE::Channel for IPC. MCE::Channel came long after making MCE::Hobo and thought to make something similar that works with Perl v5.8.1 and above.

use v5.10.1;
use MCE::Child;
 
MCE::Child->init(
    max_workers => 50,
    posix_exit  => 1,
    on_finish   => sub {
        my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @
+_;
        print "child $pid completed: $ident => ", $resp->[0], "\n";
    }
);
 
foreach my $data ( 1..2000 ) {
    MCE::Child->create( $data, sub {
        MCE::Child->yield(0.008);
        # sleep 1;  # simulate connection instantiation
        [ $data * 2 ];
    });
}
 
MCE::Child->wait_all;
[download]

Writing MCE::Child was helpful in improving MCE::Hobo and vice versa. Yeah, there are two similar modules. MCE::Child is mostly compatible with MCE::Hobo. MCE::Hobo is more compatible with threads, but requires Perl v5.10.1 minimally.