Use a dbm rather than a flat file. A good dbm can handle terabytes of data. The other thing that sounds suspect in your plan is loading the e-mail data into an in-memory hash, which obviously is limited by the amount of free memory you have. You might have to load it in chunks if you don't have enough RAM.