in reply to Re: Why should any one use/learn Perl 6?
in thread Why should any one use/learn Perl 6?

Parallelism: Doing this right in Perl 5 needs a lot of boilerplate ...

Fake news!

See MCE and MCE::Shared.

Consider the following utility script to delete records from a customer account in a global cloud services provider using their API: 

use strict; use warnings; use feature 'say'; use Data::Dumper;
use MyClient;

my $client = MyClient->new;

my $list = $client->call('ListCustomers');
$list->parse_json_payload;

say sprintf 'Start: Found %d accounts.', $list->payload->{totalCount};

my %results;

for my $account ( @{ $list->payload->{items} } ) {
    my $id = $account->{id};
    say "Deleting $id";

    my $call = $client->call('DeleteCustomer', { account_id => $id });
    say 'Status: ' . $call->status;

    $call->status == 204 ? $results{OK}++ : $results{NOT_OK}++;
}

say 'Results: ' . Dumper \%results;
__END__

[download]

Now, parallelized with MCE. Too much boilerplate? 

use strict; use warnings; use feature 'say'; use Data::Dumper;

use MyClient;
use MCE;
use MCE::Shared;

my $client = MyClient->new;

my $list = $client->call('ListCustomers');
$list->parse_json_payload;

say sprintf 'Start: Found %d accounts.', $list->payload->{totalCount};

tie my %results, 'MCE::Shared';

sub task {
    my $id = $_->{id};
    my $output  = "Deleting $id\n";

    my $call = $client->call('DeleteCustomer', { account_id => $id });
    say $output . 'Status: ' . $call->status;

    $call->status == 204 ? $results{OK}++ : $results{NOT_OK}++;
}

MCE->new( chunk_size => 1, max_workers => 7, user_func => \&task );
MCE->process( $list->payload->{items} );
MCE->shutdown;

say 'Results: ' . Dumper \%results;
__END__

[download]

Note that in the second version, we can still just use $call->status == 204 ? $results{OK}++ : $results{NOT_OK}++; even though we have 7 workers writing to the same hash.

(Also note that MCE provides comprehensive signal handling, so if you use an END block (not shown here) to print out your results hash (for example), you can kill the script eg with CTRL-C, and all workers will exit gracefully, while only the parent prints out the data so far.)

You sure as hell don't need Cuckoo, errr, notPerl "6", or anything other than Perl to write clean, fast, correct, parallel code that can be used today in performance-critical environments, aka Real Life.

Hope this helps!


The way forward always starts with a minimal test.

Replies are listed 'Best First'.
Re^3: Why should any one use/learn Perl 6?
by liz (Monsignor) on Jun 11, 2018 at 12:24 UTC
    You sure as hell don't need Cuckoo, errr, notPerl "6", or anything other than Perl to write clean, fast, correct, parallel code that can be used today in performance-critical environments, aka Real Life.

    Why can't you just be positive? Always so negative!

    I hadn't heard about MCE yet. By the looks of it, it has taken the ideas I had when I implemented the forks and ThreadPool modules to the next level. Kudos to Mario Roy.

    Now to get back to your example, but then in Perl 6: 

    use MyClient;

my $client = MyClient.new;

my $list = $client.call('ListCustomers');
$list.parse_json_payload;

say "Start: Found $list.payload.totalCount() accounts";

my %results;

for $list.payload.items -> $account {
    my $id = $account.id;
    say "Deleting $id";

    my $call = $client->call('DeleteCustomer', account_id => $id);
    say 'Status: ' . $call.status;

    $call.status == 204 ?? %results<OK>++ !! %results<NOT_OK>++;
}

say "Results: %results.perl()"

    [download]

    To make this run on multiple CPU's, one basically needs to do only one thing: prefix the for with a hyper; 

    hyper for $list.payload.items -> $account {

    [download]

    There is however one caveat: hashes in Perl 6 are not thread safe in the sense that they might lose updates when being updated from multiple threads. If you're just interested in number of successes / fails, why not make it two counters instead of elements in a hash? When updating integers from multiple threads, you should use atomic integers (otherwise you would have the same problem as with the hash elements). So the code becomes: 

    use MyClient;

my $client = MyClient.new;

my $list = $client.call('ListCustomers');
$list.parse_json_payload;

say "Start: Found $list.payload.totalCount() accounts";

my atomicint $ok;
my atomicint $not-ok;

hyper for $list.payload.items -> $account {
    my $id = $account.id;
    say "Deleting $id";

    my $call = $client->call('DeleteCustomer', account_id => $id);
    say 'Status: ' . $call.status;

    $call.status == 204 ?? atomic-inc-fetch($ok) !! atomic-inc-fetch($
+not-ok)
}

say "Results: ok = $ok, not-ok = $not-ok"

    [download]

    I've used the ASCII equivalent of the atomic increment operator, since PerlMonks does not display the unicode equivalent "++⚛️" inside code blocks properly.

    So I would argue three things here:

    1. Perl 6 has less boilerplate to begin with (no use statements needed)
    2. Perl 6 needs less boilerplate to parallelize (just adding "hyper" will do in many cases
    3. Some thought needs to go into updating data structures from multiple threads at the same time

    Note that the Perl 5 solution to updating shared data structures requires tieing and locking, neither of which is good for performance. The Perl 6 solution using atomic increment is lockless and uses hardware features of the CPU.

    Hope this clarifies.

    EDIT: fixed $results{} -> %results<> non-migrato, haj++

[reply]
[d/l]
[select]

      I'd have expected Perl 6 to have less boilerplate than Perl 5 for parallel processing. It's just that the advantage is smaller when you know about MCE than when you try to figure all of that out with Perl 5 core. So this is a Perl 6 benefit in the same order of magnitude as with objects, where I need to use Moose; in Perl 5 to get the object magic.

      Two minor nitpickings:

      • $call.status == 204 ?? $results<OK>++ !! $results<NOT_OK>++; seems somewhat Perl5ish. Shouldn't that read like that:
        $call.status == 204 ?? %results<OK>++ !! %results<NOT_OK>++;
      • You write: "I've used the ASCII equivalent of the atomic increment operator, since PerlMonks does not display the unicode equivalent "++⚛️" inside code blocks properly.". I'm glad you did, because this is another example why I don't like these unicode operators: I have no idea how to enter it into my keyboard easily, and my preferred font in my preferred editor doesn't have a glyph for it. There also seems to be another, quite similar symbol: "++⚛" which has an almost identical glyph in the textarea field into which I'm typing right now. I am pretty sure that I'll never use these symbols, and I hope I'll never need to maintain code which contains them.
[reply]
[d/l]
[select]
        ...why I don't like these unicode operators: I have no idea how to enter it into my keyboard easily...

        FWIW, those characters live in my TouchBar on my MacBook Pro.

        ...my preferred font in my preferred editor doesn't have a glyph for it...

        That's not really a Perl 6 problem. I would say then that your preferred editor will have more and more problems surviving in a more and more unicodey world. Which of course means nothing about how useful it is in its current version for you and the type of work you do with it.

        In any case, Perl 6 has ASCII variants for all of its operators, so if you don't want to use them, then that's fine too!

[reply]

      "hashes in Cuckoo are not thread safe in the sense that they might lose updates when being updated from multiple threads"

      Oh, dear. Sounds like a rather significant deficiency compared to Perl, when we are discussing concurrency. (By the way, note that MCE does not require threads to parallelize.)

      "Note that the Perl 5 solution to updating shared data structures requires tieing and locking."

      Wrong. It does not.

      Why can't you just be positive? Always so negative!

      I'm negative about your project because it is still squatting on Perl's name, duh.


      The way forward always starts with a minimal test.
[reply]
        "hashes in Cuckoo are not thread safe in the sense that they might lose updates when being updated from multiple threads" Oh, dear. Sounds like a rather significant deficiency compared to Perl, when we are discussing concurrency.

        Perl 6 has decided that it is not a good idea to shuffle the inherent problems of updating a data-structure like a hash from multiple threads at the same time, under the carpet. Tieing a hash with a Perl interface to make sure that all updates are done sequentially, is not a good step towards making a fully functional, and well performing threaded program without any bottlenecks. You, as a developer, need to be aware of the issues, and make adaptations to your program and/or the way you think about threaded programming.

        Think about writing your solutions as a pipeline, or using react whenever using an event driven model.

        In that sense, Perl 5 ithreads makes you a lazy programmer with everything being thread-local by default.

        "Note that the Perl 5 solution to updating shared data structures requires tieing and locking." Wrong. It does not.

        If I look at the code of MCE::Shared and MCE::Shared::Scalar, I do see things like a sub TIESCALAR, and &MCE::Shared::Scalar::new being bound to said TIESCALAR. That to me implies tieing. Or am I wrong?

        I'm negative about your project because it is still squatting on Perl's name, duh.

        I'm glad to hear that it's only the name you object to now.

[reply]
[d/l]
[select]
Re^3: Why should any one use/learn Perl 6?
by haj (Vicar) on Jun 08, 2018 at 13:14 UTC

    I wasn't aware of that module (MCE). Thanks for the info!

    I still prefer to refer to Perl 6 as "Camelia" :)

[reply]

      MCE is incredibly powerful, and offers a number of different models to approach parallel tasks. In many cases there is both an OO and a traditional interface. All this power and flexibility can make it difficult to know how to implement the toolset in your app. Luckily, the author marioroy can usually be summoned to these halls to provide assistance.

      (That is, I suppose, one of the pluses about the fact that MCE and MCE::Shared are so far not widely known. As you no doubt know, the authors of similarly scoped toolsets tend not to be available for one-on-one support. I don't imagine it can stay that way forever in the case of MCE either ...)


      The way forward always starts with a minimal test.
[reply]

In Section Seekers of Perl Wisdom