Having done processing of flat files in the way you have to, I can tell you that it's far, far easier, and often faster, to use a DB. Throughput is normally better, and having the data queryable in an easy way gives you the opportunity to analyze it in ways you may not have thought of, or though of but discarded because it was infeasable because of processing concerns.