If you do need to improve runtime you should try to be smart rather than clever. A smart solution is likely to approach the problem from an entirely different direction. A clever solution is likely to be a nasty mess arrived at by iterative tweaking the code to squeeze the last drop of performance out of a sub-optimum algorithm.

-- GrandFather

Love it! ... so much that I could not restrain myself from adding it to on Code Optimization and Performance References. :-)

BTW, my favourite quote from that node is from Michael Abrash:

Without good design, good algorithms, and complete understanding of the program's operation, your carefully optimized code will amount to one of mankind's least fruitful creations -- a fast slow program.

👁️🍾👍🦟

Do you have a benchmarked application where the benchmark is showing that foreach/grep,map is the specific bottleneck and causing tens of seconds of extra run time?

I am processing some medium-sized files (1.6 MB to 2.5 MB etc) and notice that the run time goes up from ~ 3 s to 5+ s with grep and map compared to foreach. However, I see now that I can tune things better.

I suggest you create and post a working program using Perl's core Benchmark module that anyone can just cut and paste and run for themselves. Apart from excellent learning for you, doing that should also provoke more helpful responses from the monks.

Some example PM nodes that have taken that approach:

• Re: Confused by RegEx count by choroba
• Fastest way to lookup a point in a set
• Re^3: looping efficiency (Benchmark Example)
• How to do popcount (aka Hamming weight) in Perl (popcount References)
• Re^2: Speed comparison of foreach vs grep + map by ikegami - example later in this thread Benchmarking map, grep and for
• Re^2: Perl at Rosetta Code, with one particular example by anonymonk - comparing his solution with tybalt89's solving this rosetta code

See also

• perlperf - Perl Performance and Optimization Techniques (perldoc)
• on Code Optimization and Performance References

Updated: Added more example PM nodes and See also section.

👁️🍾👍🦟

Thanks, yet again. The Benchmark qw(:hireswallclock) module really shows the difference between the two code segments:

        my $t0 = Benchmark->new;
        # build TOC
                 
        foreach my $h (split(/\n/, $page)) {
            if ($h =~ m/^={2,3}\s/) {
                push(@toc, $file."\t".$h);
            }
        }
        my $t1 = Benchmark->new;
        my $td = timediff($t1, $t0);
        print "the code took:",timestr($td),"\n";
[download]

and

        my $t0 = Benchmark->new;
        # build TOC
        push(@toc,
              map(m/^={2,3}\s/ ? $file."\t".$_ : (),
                   split(/\n/, $page) ) );  
        my $t1 = Benchmark->new;
        my $td = timediff($t1, $t0);
        print "the code took:",timestr($td),"\n";
[download]

With the data sets I have, the first one is about a second faster per subset of data. I hope those two segments are similar enough to compare. I don't think I can use substr() because the pieces sought are of variable length and alternations slows it way down.

        foreach my $h (split(/\n/, $page)) {
            if (substr($h,0,3) eq '== ' 
                 or substr($h,0,4) eq '=== ') {
                push(@toc, $file."\t".$h);
            }
        }
[download]

That adds up to 1.5 seconds.

For what it's worth, the data is by that point markdown text — and a lot of it. The sources are OpenDocument Format word processing files. I got the first 90% of the task done in pure XML using XML::Twig in two afternoons, thanks to advice here. It's the second 90%: dealing with nested lists, cross-references, and individual chapters which is taking even more time. It seems that few are using sections within their ODF files, and so "chapters" end up being all part of the same parent element. Parkinson's law applies as well.