Why do you call map { $_->[0] } on every element if you just want one?

Note that the map EXPR, syntax of map is faster than the map BLOCK syntax.

And then there's the issue that you've taken an O(N) problem and made it O(N log N) by using sort, which means it doesn't scale as well.