Well, it was already given that storing it all in an array was taking too much memory. Given that a hash uses even more overhead than an array, your solution only wins if there are many duplicates. And while an MD5 hash may only be 32 bits (4 bytes), a single Perl scalar already takes at least 24 bytes. Plus two handfuls of bytes for being a hash entry. And the overhead of the hash itself.

Abigail