in reply to Improving the efficiency of code when processed against large amount of data

It has been my experience that the best way to improve the performance of code is to profile it. Find out where the most amount of time is spent. If the majority of the time is spent in file I/O, then you can't do much because the speed is limited by your seek/access times. However, by profiling, you can find out which lines can potentially be sped up. You could also consider forking off and running process in parallel (using Parallel::ForkManager). Back to profiling though...check out Devel::SmallProf or Devel::FastProf for line and subroutines profiling. Here is also a little HOWTO that I found useful for getting started (if it is your first attempt at code profiling): http://www.ddj.com/184404580.
  • Comment on Re: Improving the efficiency of code when processed against large amount of data