You don't say what you're doing with all that data, which would help you get a useful answer. For many purposes, sticking them into a database of some kind would be the thing to do.