I think you should consider using BerkeleyDB's array-like interface, or some similar dbm solution. It does a good job with the memory management for you, and can be hidden behind a tied array if you like.