How do I get better at thinking about optimizing my Perl?

amarquis has asked for the wisdom of the Perl Monks concerning the following question:

(Besides practice)

Specifically, I feel like I'm missing intuition in my optimization procedure. Here's what I do:

Decide what is the bottleneck either via profiling or guesswork.
Pick the specific operations that are causing the most grief, and think up better ways to accomplish the same goals.
Try some solutions, benchmarks are often involved.

Step 2 has some serious voodoo in it. Not good voodoo, bad voodoo. When I program in C, my guesses as to what needs to change and how to change it are pretty good. When working with Perl, I'm often shocked by which solutions perform faster, which is telling me that I've got a really poor mapping in my mind between operations in Perl and how much effort Perl exerts to accomplish them.

I don't feel like practice is teaching me much in this regard; I just end up confused as to why X is faster than Y. When perl is not quite fast enough's "What causes slowness in Perl" section has been a good resource for me, but I'm looking for more. Are there other resources out there (I've got the camel and the cookbook on my desk, as far as texts go) that would help wrap my head around this better? Reading posts around PerlMonks that optimize others' code make me really wish I was better at this.

Thanks in advance, and have a great thanksgiving if you are a thanksgiving-celebrating-monk.

Comment on How do I get better at thinking about optimizing my Perl?

Replies are listed 'Best First'.
Re: How do I get better at thinking about optimizing my Perl? by perrin (Chancellor) on Nov 21, 2007 at 16:21 UTC
It's no different from C really. To make things faster, do less work. The big gains come from eliminating I/O or improving your algorithm. Sometimes you can cache things. Sometimes you can write better SQL. The only thing that gets very Perl-specific is learning which modules are faster (Storable beats Data::Dumper, the shared memory cache modules are slow) and some API specifics for common modules like DBI (prepare_cached, bind_cols). One thing that I do consider a bit of a black art is regex optimization, but you'll find there are a number of tools to help with that, and some very knowledgeable people on PerlMonks who I often turn to with my regex problems.	[reply]
Re: How do I get better at thinking about optimizing my Perl? by perlfan (Parson) on Nov 21, 2007 at 15:25 UTC
Start by running your code using `perl -d:DProf`, then run `dprofpp`. This will tell you where you're spending most of your time. There is a ton of stuff out there on profiling perl.	[reply] [d/l] [select]
Re: How do I get better at thinking about optimizing my Perl? by dragonchild (Archbishop) on Nov 21, 2007 at 16:44 UTC
A Perl program that doesn't interact with outside entities (database, filesystem, networked thingy, etc) is very rare. In every case I've ever seen one, they have never taken more than a second or two to run. Thus, they didn't need optimizing. Nearly every Perl program that interacts with an outside entity is almost always bound by the usage of outside entity. For example, optimizing your database tables and the SQL you use will almost always yield enough performance benefit to satisfy. This leads us to the understanding that the primary things that need optimizing are the ways we interact with the outside world. Thus, a while-loop to iterate over a file is (almost) always better than the equivalent foreach-loop. The one real exception to this are regexes. But, in 12 years, I haven't found a use for the kind of regex that needs optimization. I don't use backtracking, lookaheads (except rarely), and all that other black magic. As for webapps, fcgi and mod_perl are all you need. My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re: How do I get better at thinking about optimizing my Perl? by olus (Curate) on Nov 21, 2007 at 17:28 UTC
It is my belief that you could get better by not considering guesswork if you are serious into optimization. You must consider facts and you must work on what you can control, so rethinking your algorithm would be the first approach. Then you can benchmark your different solutions and choose the one that performs better. After choosing the best algorithm you can dwell upon turning your code unreadable by using some of the good old techniques that you must be familiar with from your C background, such as lookup tables or, even deeper, loop unroling, less function calls .... Your desk is lacking an algorithms book :) -- olus	[reply]