(or, maybe an array of hashes?) might be the best fundamental representation in your program. Because, what you are basically trying to do in this single statement ... what you are probably asking Perl to do ... is to construct and then sort an array on-the-fly.
Well, that's a very expensive thing to do, especially when you don't realize that you're doing it.