in reply to Optimizing Iterating over a giant hash
Why are you not using a database? You may find Re: Data Structure Question a useful starting point.