Perl hashes are blazingly fast, if they fit into memory. There's not much room for optimization there.

Use Devel::NYTProf to find where your script actually spends its time.