Re^2: IPC::Msg Fork queue

Hi trippledubs,

MCE can process a sequence of numbers efficiently. E.g. 1 to MAX_ITEMS. The bounds_only option tells MCE to pass the next starting and ending elements only. Hence boundaries only. IPC occurs once per each chunk_size.

#!/usr/bin/env perl
use strict;
use warnings;
use MCE;

use constant MAX_ITEMS => 200000;

my $n_workers = shift // do {
    die "usage: $0 n_workers\n";
};

sub isPrime {
    my $num = shift;
    return 1 if ($num < 4);
    return 0 if ($num %2 == 0);
    for (my $i=3; $i <= sqrt($num); $i+=2) {
        return 0 if $num % $i == 0;
    }
    return 1;
}

sub task {
    my ( $mce, $seq, $chunk_id ) = @_;
    for my $data ( $seq->[0] .. $seq->[1] ) {
        print "Prime: $data\n" if isPrime($data);
    }
}

MCE->new(
    max_workers => $n_workers,
    sequence    => [ 1, MAX_ITEMS ],
    bounds_only => 1,
    chunk_size  => 200,
    user_func   => \&task,
)->run();
[download]

Regards, Mario

Comment on Re^2: IPC::Msg Fork queue Download Code

Replies are listed 'Best First'.
Re^3: IPC::Msg Fork queue by marioroy (Prior) on Feb 24, 2019 at 05:28 UTC
Hi again, If needed, workers may write output orderly and efficiently via MCE::Relay simply by passing the init_relay option. The value to init_relay is not used in the demonstration. Just the block to MCE::relay which runs serially and orderly. #!/usr/bin/env perl use strict; use warnings; use MCE; use constant MAX_ITEMS => 200000; $\| = 1; # necessary when workers output orderly my $n_workers = shift // do { die "usage: $0 n_workers\n"; }; sub isPrime { my $num = shift; return 1 if ($num < 4); return 0 if ($num %2 == 0); for (my $i=3; $i <= sqrt($num); $i+=2) { return 0 if $num % $i == 0; } return 1; } sub task { my ( $mce, $seq, $chunk_id ) = @_; my $output = ''; for my $data ( $seq->[0] .. $seq->[1] ) { $output .= "Prime: $data\n" if isPrime($data); } MCE::relay { print $output }; } MCE->new( max_workers => $n_workers, sequence => [ 1, MAX_ITEMS ], bounds_only => 1, chunk_size => 200, user_func => \&task, init_relay => 1, )->run(); [download] Regards, Mario	[reply] [d/l]
Re^3: IPC::Msg Fork queue by trippledubs (Deacon) on Feb 26, 2019 at 15:32 UTC
Love it! What I wanted to get across is multiple workers operating, enqueing and dequeing, inside same queue. Like Thread::Queue or MCE::Queue. Checking if a number is prime, probably not the best example, but easy to understand. That is not much different than what Parallel:ForkManager can do. Re: Sum to 100 at Rosetta Code is the better example. If you can get your problem into a queue, and keep possible solutions inside the queue, then that design can be made to be concurrent. The best example would probably be an Evolutionary Algorithm or something like that. Queues are awesome anyways, fun to say, rhyme with a lot of other words, lot of vowels.. Not communist and unfair like a stack. They are primitives that can be used as building blocks, observable, core perl, been around forever, reentrant, configurable, and persistent. Seems to scale well and does priorities too. And with priorities comes the ability to order the output and sort. API stinks, but documented well. From what I read just a linked list inside the kernel. But it doesn't do chunking for you like MCE, well it doesn't do a lot of things MCE does for you, which as you illustrate is a big bottleneck.	[reply]