I will hook up Time::HiRes and start timing the loops

You might want to look up Devel::Dprof instead. It's part of the core distribution, and it gives you a nice breakdown of where the time goes. In a nutshell:

 perl -d:Dprof your_script

 # wait 160 seconds or whatever...
 
 dprofpp
[download]

(the "dprofpp" tool should be in the same path where perl was installed). To get the best use out of it, you might want to split up your code to put one or more of the inner loops into separate subroutines, to get better granularity on the timing analysis.

(While breaking things up that way, who knows... you might figure out some other way to do it -- then you can use Benchmark to compare which approach is best, in case it's not immediately obvious. :)