My goal was to refactor the OP's code in a straight-forward way, with the intent of being instructive - there are of course imperfections .
When coding this myself, I would be thinking about functions in List::Util and passing references to array instead of an array itself. But that is not what I thought my job here was. My intent was to present easy to understand code. But even so, I figure that the OP will have some questions about it.