Try presizing the hash - keys %lookup = 160_000;

If it is hash collisions, this might solve the problem.

dominus has an interesting bit at perl.plover.com called When Hashes Go Wrong.

Update: Meant to ask you to "print scalar %lookup;" after all is done. scalar %hash will give you the number of used buckets / number of allocated buckets. If the number of used buckets is low (like 1/16) all your hash items have been put in the same bucket!