Re^2: An efficient way to parallelize loops

Hi moritz

First off, thanks for you kind reply.

Your first point, unluckely is not applicable, due to the fact that depending on the element in @{$categories{$k}->{traces}} matched, the sum below will be applied to a different element in the @lista array.

I was trying to find other solutions, actually, to read the file. Does the while loop run faster than doing a foreach loop, or putting the whole file into an array, and then reading the array line by line?

Thanks for the help!

Comment on Re^2: An efficient way to parallelize loops

Replies are listed 'Best First'.
Re^3: An efficient way to parallelize loops by moritz (Cardinal) on Jun 01, 2010 at 09:36 UTC
Your first point, unluckely is not applicable, due to the fact that depending on the element in @{$categories{$k}->{traces}} matched, the sum below will be applied to a different element in the @lista array. I don't understand what you're saing. The code you showed us changes neither `%categories` nor `$k`. Why should anything change if you do the lookup outside the loop? `my @a = @{$categories{$k}->{traces}; OUTER: while( my $line = <GZIP> ) { for ( my $i = 0; $i < @a }; $i++ ) { if ( $line =~ /^($a[$i]->{regex})/ ) { my @lista = split /;/, $line; $A += $lista[$a[$i]->{calc}}[0]]; $B += $lista[$a[$i]->{calc}}[1]]; $C += $lista[$a[$i]->{calc}}[2]]; next OUTER; } } }` [download] This should do exactly the same as your code, only more efficient. The version with `while` is at least as fast as the version with `for`, and uses much less memory. Still I'd like to emphasize my first point again: Benchmark and profile before starting to optimize (and before even thinking of parallelization).	[reply] [d/l] [select]
Re^4: An efficient way to parallelize loops by Deus Ex (Scribe) on Jun 01, 2010 at 10:18 UTC
Dammit! I misunderstood what you wrote before: i had intended you suggested to move the whole "if" statement outside of the loop, not the lookup for the variables' values in the statement. Sorry! You were right. I'll also do the profiling, to see where the bottlenecks are. Thank you very much again!	[reply]
Re^4: An efficient way to parallelize loops by Deus Ex (Scribe) on Jun 01, 2010 at 13:23 UTC
Do you think that moving outside the loop the split() would be useful? I also thought to do like this: `open(GZIP, "<:gzip", "$path") or die "$!\n"; my @a = @{$categories{$k}->{traces}}; LOOP: while ( my $line = <GZIP>) { my @lista = split /;/, $line; my $head = $lista[0]; for ( my $i = '0'; $i < scalar @a; $i+ ++ ) { if ( $head =~ /^($a[$i]->{rege +x})$/ ) { $A += $lista[$a[$i]->{ +calc}[0]]; $B += $lista[$a[$i]->{ +calc}[1]]; $C += $lista[$a[$i]->{ +calc}[2]]; next LOOP; } } } close(GZIP);` [download] I put the first element of the @lista array as the pattern to test, cause it is there; no need to test the whole line.	[reply] [d/l]