in reply to Re^3: Parrot, threads & fears for the future.
in thread Parrot, threads & fears for the future.

What if there was a pragma for enabling parallelization? It could be use parallel, in the spirit of integer. 
my @list = ('aaaa' .. 'zzzz');
use parallel 'map'; # Just for maps
my @ordlist = map ord, @list;

# Defaults only
# Probably something like any, all, map, grep, and maybe even sort
use parallel;

# Sort by last letter in parallel
@list = sort {($a =~ /\w+(\w)/) <=> ($b =~ /\w+(\w)/)} @list;

# This subroutine can operate in parallel
use parallel 'foreach';
foo($_) foreach @list;

no parallel; # disable everywhere else

[download]

Replies are listed 'Best First'.
Re^5: Parrot, threads & fears for the future.
by Jenda (Abbot) on Oct 26, 2006 at 10:34 UTC

    I think it would be safer to have a "parallel map" function instead. In either way the problem is that people would not normaly think neither about adding the pragma nor using the parallel map. Plus there is another problem. The question is not just whether perl may execute the map in parallel, but also in how many chunks. You would of course not want to create a thread for each and every element of the array/list especially if it's huge and the operation is simple. The number of threads you will want to use depends on the complexity and nature of the action you want to do with the elements - you will want more threads if you will spend most time waiting for a network response, you will want less if it's just computation ... So without knowing what's the operation, the implementation of the mapp will not know how to parallelize the operation to achieve good performance. So even though mapp is a nice interface to parallelization, you will want to be able to give it a few more hints to ensure it works well.

    Jenda
    Support Denmark!
    Defend the free world!

[reply]
      I think that a compile-time switch that sets the maximum number of threads would be a good solution. This way, you could compile Perl with support for 8 threads on an 8 core processor, and have it be used completely.

      Each thread could get one item to start with, and as it finishes, it could get an additional item. If it's just running a built-in function on the list, then each thread could be given 1/n of the list (where n is the number of threads). Otherwise, Perl could time the items and detect about how many would be good to send based on the time.

      Perl would still need to get data back from the threads to give back to the core, so sending data shouldn't be a problem either. But I've never used threads, so I may be off.

[reply]

        I don't think a compile time switch is a good solution, that's far too restricitve. Especially since quite a few people do not compile their perl!

        Besides what you say assumes that it's always OK to use up all the power of the machine and also it's the way you would want to use the paralelization for the computationaly expensive tasks. In case the operation you want to do with the element of the list spends more time waiting for some data then you will want to use more threads. Eg. if you wanted to test the conectivity to the computers specified in the list you will spend most time waiting for the pings to return or time out, not computing anything. So you would want to create quite a few more threads than if you just need to use all the CPUs your box has.

      • Jenda
        Support Denmark!
        Defend the free world!

[reply]

In Section Meditations