You are probably simply running out of RAM. As others have discussed, Perl hashes should usually be well behaved even when they get really huge. However, it is my experience that running out of RAM means that your code slows down exponentially as its need for RAM grows.

One exception is if you can localize RAM access so that items you access at nearby times are also nearby in RAM (that is, clumped together in clusters; that is, only on a few pages of VM not each on a different page of VM). For example, looping over 2-D arrays in FORTRAN or C row-by-row can be much faster than column-by-column.

But Perl makes heavy use of pointers and so really large Perl arrays don't usually lead to very localized memory use even when you stick to one part of the array. And really large Perl hashes jump all over in RAM.

You could try tieing the hash to a DBM file. This will make the initial hash accesses quite a bit slower, but will prevent the exponential slow-down as the hash gets really large and so may well make your code run much faster in this specific case.

                - tye