Parallelism: Doing this right in Perl 5 needs a lot of boilerplate ...

Fake news!

See MCE and MCE::Shared.

Consider the following utility script to delete records from a customer account in a global cloud services provider using their API:

use strict; use warnings; use feature 'say'; use Data::Dumper;
use MyClient;

my $client = MyClient->new;

my $list = $client->call('ListCustomers');
$list->parse_json_payload;

say sprintf 'Start: Found %d accounts.', $list->payload->{totalCount};

my %results;

for my $account ( @{ $list->payload->{items} } ) {
    my $id = $account->{id};
    say "Deleting $id";

    my $call = $client->call('DeleteCustomer', { account_id => $id });
    say 'Status: ' . $call->status;

    $call->status == 204 ? $results{OK}++ : $results{NOT_OK}++;
}

say 'Results: ' . Dumper \%results;
__END__
[download]

Now, parallelized with MCE. Too much boilerplate?

use strict; use warnings; use feature 'say'; use Data::Dumper;

use MyClient;
use MCE;
use MCE::Shared;

my $client = MyClient->new;

my $list = $client->call('ListCustomers');
$list->parse_json_payload;

say sprintf 'Start: Found %d accounts.', $list->payload->{totalCount};

tie my %results, 'MCE::Shared';

sub task {
    my $id = $_->{id};
    my $output  = "Deleting $id\n";

    my $call = $client->call('DeleteCustomer', { account_id => $id });
    say $output . 'Status: ' . $call->status;

    $call->status == 204 ? $results{OK}++ : $results{NOT_OK}++;
}

MCE->new( chunk_size => 1, max_workers => 7, user_func => \&task );
MCE->process( $list->payload->{items} );
MCE->shutdown;

say 'Results: ' . Dumper \%results;
__END__
[download]

Note that in the second version, we can still just use $call->status == 204 ? $results{OK}++ : $results{NOT_OK}++; even though we have 7 workers writing to the same hash.

(Also note that MCE provides comprehensive signal handling, so if you use an END block (not shown here) to print out your results hash (for example), you can kill the script eg with CTRL-C, and all workers will exit gracefully, while only the parent prints out the data so far.)

You sure as hell don't need Cuckoo, errr, notPerl "6", or anything other than Perl to write clean, fast, correct, parallel code that can be used today in performance-critical environments, aka Real Life.

Hope this helps!

You sure as hell don't need Cuckoo, errr, notPerl "6", or anything other than Perl to write clean, fast, correct, parallel code that can be used today in performance-critical environments, aka Real Life.

Why can't you just be positive? Always so negative!

I hadn't heard about MCE yet. By the looks of it, it has taken the ideas I had when I implemented the forks and ThreadPool modules to the next level. Kudos to Mario Roy.

Now to get back to your example, but then in Perl 6:

use MyClient;

my $client = MyClient.new;

my $list = $client.call('ListCustomers');
$list.parse_json_payload;

say "Start: Found $list.payload.totalCount() accounts";

my %results;

for $list.payload.items -> $account {
    my $id = $account.id;
    say "Deleting $id";

    my $call = $client->call('DeleteCustomer', account_id => $id);
    say 'Status: ' . $call.status;

    $call.status == 204 ?? %results<OK>++ !! %results<NOT_OK>++;
}

say "Results: %results.perl()"
[download]

To make this run on multiple CPU's, one basically needs to do only one thing: prefix the for with a hyper;

hyper for $list.payload.items -> $account {
[download]

There is however one caveat: hashes in Perl 6 are not thread safe in the sense that they might lose updates when being updated from multiple threads. If you're just interested in number of successes / fails, why not make it two counters instead of elements in a hash? When updating integers from multiple threads, you should use atomic integers (otherwise you would have the same problem as with the hash elements). So the code becomes:

use MyClient;

my $client = MyClient.new;

my $list = $client.call('ListCustomers');
$list.parse_json_payload;

say "Start: Found $list.payload.totalCount() accounts";

my atomicint $ok;
my atomicint $not-ok;

hyper for $list.payload.items -> $account {
    my $id = $account.id;
    say "Deleting $id";

    my $call = $client->call('DeleteCustomer', account_id => $id);
    say 'Status: ' . $call.status;

    $call.status == 204 ?? atomic-inc-fetch($ok) !! atomic-inc-fetch($
+not-ok)
}

say "Results: ok = $ok, not-ok = $not-ok"
[download]

I've used the ASCII equivalent of the atomic increment operator, since PerlMonks does not display the unicode equivalent "++⚛️" inside code blocks properly.

So I would argue three things here:

1. Perl 6 has less boilerplate to begin with (no use statements needed)
2. Perl 6 needs less boilerplate to parallelize (just adding "hyper" will do in many cases
3. Some thought needs to go into updating data structures from multiple threads at the same time

Note that the Perl 5 solution to updating shared data structures requires tieing and locking, neither of which is good for performance. The Perl 6 solution using atomic increment is lockless and uses hardware features of the CPU.

Hope this clarifies.

EDIT: fixed $results{} -> %results<> non-migrato, haj++

I'd have expected Perl 6 to have less boilerplate than Perl 5 for parallel processing. It's just that the advantage is smaller when you know about MCE than when you try to figure all of that out with Perl 5 core. So this is a Perl 6 benefit in the same order of magnitude as with objects, where I need to use Moose; in Perl 5 to get the object magic.

Two minor nitpickings:

• $call.status == 204 ?? $results<OK>++ !! $results<NOT_OK>++; seems somewhat Perl5ish. Shouldn't that read like that:
  $call.status == 204 ?? %results<OK>++ !! %results<NOT_OK>++;
• You write: "I've used the ASCII equivalent of the atomic increment operator, since PerlMonks does not display the unicode equivalent "++⚛️" inside code blocks properly.". I'm glad you did, because this is another example why I don't like these unicode operators: I have no idea how to enter it into my keyboard easily, and my preferred font in my preferred editor doesn't have a glyph for it. There also seems to be another, quite similar symbol: "++⚛" which has an almost identical glyph in the textarea field into which I'm typing right now. I am pretty sure that I'll never use these symbols, and I hope I'll never need to maintain code which contains them.

"hashes in Cuckoo are not thread safe in the sense that they might lose updates when being updated from multiple threads"

Oh, dear. Sounds like a rather significant deficiency compared to Perl, when we are discussing concurrency. (By the way, note that MCE does not require threads to parallelize.)

"Note that the Perl 5 solution to updating shared data structures requires tieing and locking."

Wrong. It does not.

Why can't you just be positive? Always so negative!

I'm negative about your project because it is still squatting on Perl's name, duh.



The way forward always starts with a minimal test.

I wasn't aware of that module (MCE). Thanks for the info!

I still prefer to refer to Perl 6 as "Camelia" :)

MCE is incredibly powerful, and offers a number of different models to approach parallel tasks. In many cases there is both an OO and a traditional interface. All this power and flexibility can make it difficult to know how to implement the toolset in your app. Luckily, the author marioroy can usually be summoned to these halls to provide assistance.

(That is, I suppose, one of the pluses about the fact that MCE and MCE::Shared are so far not widely known. As you no doubt know, the authors of similarly scoped toolsets tend not to be available for one-on-one support. I don't imagine it can stay that way forever in the case of MCE either ...)



The way forward always starts with a minimal test.

I am also accumulating a list of things I would prefer not to see in productive code. I'll most certainly never use these myself.

Single quotes and minus signs in variable names.

Letters with diacritics in variable names.

Literals like ½ for numbers. Unicode operators which I don't see on my keyboard.

Quoting functions with three or more letters. qqw qww qqww.

About 3.5 years ago, I posted Quoting on Steroids, which hopefully made some sense to some. :-)