I'll not even try to understand what your code does (and other already provided some material for you on that), but here are some general guidelines:

1. You should run a profiler over it: see Devel::NYTProf and identify the areas that are using more time/CPU so you can start with them.
2. Since you are working inside loops, check out if you can get ride of them. If not, check some algorithms like Schwartzian_transform.
3. Benchmark your code before and after the changes you implement to see if they are effective. See Dumbbench.