in reply to Verifying data in large number of textfiles

Perhaps do some extra work while you're parsing the data. As mentioned above, check white space, numbers, id, dates.

Is there anything in particular that worries you? Which part of the data are you particularly interested in?

I would consider generating additional reports.

Also, if this would only be run occasionally I wouldn't consider 5000 files a problem. I would be tempted to get in and have a good look!

As usual, it depends, but I think you could reassure yourself more on that first pass.

  • Comment on Re: Verifying data in large number of textfiles