in reply to storing hash in temporary files to save memory usage
Have you tried tieing the hashes to files (eg. with Tie::Hash)? If so, please elaborate on precisely how this failed to meet your needs.