in reply to Parse data into large number of output files.
The real RDMBS suggestion is probably going to be the best way, but if you want to give it a go with multiple files check out FileCache.