You can only transparently paralelize map{} if the executed block is side-effect-free. Perl is far too dynamic for the compiler/optimizer to safely check that and you might very well want to have side-effects in your map{}s so you'd have to have two map{}s, a paralellisable one and a garanteed-to-be-serial one. It's one thing to paralelize a purely functional, side-effect free language and to paralelize something that may do whatever it bloody wishes. Everything comes for a prize, even freedom.

Sure - but I'm not talking about doing that for Perl5, but postulating it for Perl6, where such stuff is still possible.

>> You can only transparently paralelize map{}

I think that's exactly what I'm seeking to do in Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)

Here "hashmap" would appear to be the parallelisable function, which is (I hope) side effect free. I have a similar function I'm playing with (not mentioned in the post) which I'm calling vector_reduce. This can't be parallelised.

The two functions together seem to be the conceptual core, or something close by, of MapReduce, made famous by google, and newly on cpan.

What if there was a pragma for enabling parallelization? It could be use parallel, in the spirit of integer.

my @list = ('aaaa' .. 'zzzz');
use parallel 'map'; # Just for maps
my @ordlist = map ord, @list;

# Defaults only
# Probably something like any, all, map, grep, and maybe even sort
use parallel;

# Sort by last letter in parallel
@list = sort {($a =~ /\w+(\w)/) <=> ($b =~ /\w+(\w)/)} @list;

# This subroutine can operate in parallel
use parallel 'foreach';
foo($_) foreach @list;

no parallel; # disable everywhere else
[download]

I think it would be safer to have a "parallel map" function instead. In either way the problem is that people would not normaly think neither about adding the pragma nor using the parallel map. Plus there is another problem. The question is not just whether perl may execute the map in parallel, but also in how many chunks. You would of course not want to create a thread for each and every element of the array/list especially if it's huge and the operation is simple. The number of threads you will want to use depends on the complexity and nature of the action you want to do with the elements - you will want more threads if you will spend most time waiting for a network response, you will want less if it's just computation ... So without knowing what's the operation, the implementation of the mapp will not know how to parallelize the operation to achieve good performance. So even though mapp is a nice interface to parallelization, you will want to be able to give it a few more hints to ensure it works well.

Jenda


Support Denmark! Defend the free world!