in reply to An efficient way to parallelize loops

I don't know how complex the $categories{$k}->{traces}[$i]->{regex} patterns are, but due to you looping over a set of patterns for each line, you do a regexp compile for each inner loop. You may want to precompile the patterns (note that even if $categories{$k}->{traces}[$i]->{regex} is a qr// construct, the fact you are using it inside a larger pattern (the anchor and the parens) makes that it gets stringified and recompiled each time. Alternatively, you may swap the inner and outer loop (that is, for each pattern, loop over the file) - but you'll have to do some benchmarking, whether or not it's faster depends on all kinds of factors.

And as others already have pointed out - use a while loop when iterating over the handle instead of a foreach.

Replies are listed 'Best First'.
Re^2: An efficient way to parallelize loops
by Deus Ex (Scribe) on Jun 01, 2010 at 10:31 UTC

    Hi javaFan

    Thanks for your help. The patterns are not complex, but long, since they're made of all the elements of long arrays, separated by "|" (with the intent of alternatively match different patterns).

    As you suggested before, I already precompile the regex with qr// operator, which already sped up the code.

    I'll try to swap the loops, though, to see if there's any improvement, and I'll do some profiling as well.

    Many thanks again for your help!

[reply]
      The patterns are not complex, but long, since they're made of all the elements of long arrays, separated by "|" (with the intent of alternatively match different patterns).

      Then you'll likely benefit from running perl 5.10 or newer, since it implements a Trie optimization for alternations of literal patterns. If the arrays are really huge, you could increase the value of ${^RE_TRIE_MAXBUF} to make them all fit into the same trie.

      Perl 6 - links to (nearly) everything that is Perl 6.
[reply]

        I've been working on a machine which is not run by me. There's only a 5.8 perl environement, and no way to get it upgraded.

        Thanks for the hint though.

[reply]

In Section Seekers of Perl Wisdom