You might also look into something like BerkeleyDB and keep the collated results in a hash-on-disk instead of in RAM. There'll be a bit more overhead but it might be the trick to go from "MOMMY MOMMY MAKE THE THRASHING STOP" to "working acceptably fast".

Also don't discount just getting more RAM. It's probably going to be much cheaper and perform better than you spending significant time working around what can be solved by around $100-200 worth of DIMM or SIMM or whatever they're calling them these days.