|
in reply to IPC::Msg Fork queue
Wow SysV is cool to learn. There is more to it than I thought though. Two things operating out of the same queue will not work without syncing. I *think* the code I wrote is okay since the queue is filled up and then processed, but SysV IPC must be some of the oldest code still in Linux. Of the three books I have that reference it, seems it was still shiny back in 1986, discouraged in 2005, and downright hated by everyone in 2012. First sentence in Programming Perl about System V IPC: "Everyone hates System V IPC." I didn't know enough about it to hate it yet. You'd think a queue implementation with fork would be relatively not hard to do. Totally new to me I thought I was breaking new ground about to dance in the graveyard of conventional wisdom -- not so much. You need something like IPC::Semaphore to control access, and then you have to clean that up as well, which presents problems because you want to delete the queue when it is empty, but delete the queue too soon, and it becomes quite difficult to lock. Delete it too late and it's just hanging around somewhere in memory forever. And it doesn't do chunking, nobody wants to dequeue just one item. Okay.. I'm starting to get it now -- at the end of the CPAN road is MCE or you could go cooperative multitasking with Coro.. gotta go but i wanted to post this now
Update 2019-02-04 - Wait properly and destroy the queue from the parent after all children have been reaped |
Re^2: IPC::Msg Fork queue
by marioroy (Prior) on Feb 24, 2019 at 05:08 UTC
|
Hi trippledubs,
MCE can process a sequence of numbers efficiently. E.g. 1 to MAX_ITEMS. The bounds_only option tells MCE to pass the next starting and ending elements only. Hence boundaries only. IPC occurs once per each chunk_size.
#!/usr/bin/env perl
use strict;
use warnings;
use MCE;
use constant MAX_ITEMS => 200000;
my $n_workers = shift // do {
die "usage: $0 n_workers\n";
};
sub isPrime {
my $num = shift;
return 1 if ($num < 4);
return 0 if ($num %2 == 0);
for (my $i=3; $i <= sqrt($num); $i+=2) {
return 0 if $num % $i == 0;
}
return 1;
}
sub task {
my ( $mce, $seq, $chunk_id ) = @_;
for my $data ( $seq->[0] .. $seq->[1] ) {
print "Prime: $data\n" if isPrime($data);
}
}
MCE->new(
max_workers => $n_workers,
sequence => [ 1, MAX_ITEMS ],
bounds_only => 1,
chunk_size => 200,
user_func => \&task,
)->run();
Regards, Mario | [reply]
[d/l] |
|
|
#!/usr/bin/env perl
use strict;
use warnings;
use MCE;
use constant MAX_ITEMS => 200000;
$| = 1; # necessary when workers output orderly
my $n_workers = shift // do {
die "usage: $0 n_workers\n";
};
sub isPrime {
my $num = shift;
return 1 if ($num < 4);
return 0 if ($num %2 == 0);
for (my $i=3; $i <= sqrt($num); $i+=2) {
return 0 if $num % $i == 0;
}
return 1;
}
sub task {
my ( $mce, $seq, $chunk_id ) = @_;
my $output = '';
for my $data ( $seq->[0] .. $seq->[1] ) {
$output .= "Prime: $data\n" if isPrime($data);
}
MCE::relay { print $output };
}
MCE->new(
max_workers => $n_workers,
sequence => [ 1, MAX_ITEMS ],
bounds_only => 1,
chunk_size => 200,
user_func => \&task,
init_relay => 1,
)->run();
Regards, Mario
| [reply]
[d/l] |
|
|
Love it! What I wanted to get across is multiple workers operating, enqueing and dequeing, inside same queue. Like Thread::Queue or MCE::Queue. Checking if a number is prime, probably not the best example, but easy to understand. That is not much different than what Parallel:ForkManager can do. Re: Sum to 100 at Rosetta Code is the better example. If you can get your problem into a queue, and keep possible solutions inside the queue, then that design can be made to be concurrent. The best example would probably be an Evolutionary Algorithm or something like that. Queues are awesome anyways, fun to say, rhyme with a lot of other words, lot of vowels.. Not communist and unfair like a stack.
They are primitives that can be used as building blocks, observable, core perl, been around forever, reentrant, configurable, and persistent. Seems to scale well and does priorities too. And with priorities comes the ability to order the output and sort. API stinks, but documented well. From what I read just a linked list inside the kernel. But it doesn't do chunking for you like MCE, well it doesn't do a lot of things MCE does for you, which as you illustrate is a big bottleneck.
| [reply] |
|
|
