I have a consistent 3 second improvement

3 seconds saved on 463 seconds runtime hardly seems worth the effort. But you were never going to save much on the call overhead of subs that iterate over large arrays internally.

You can easily save more than that by some slight rearrangements of the sub internals.

And far more by using List::Util.