Ok, so give us a sample of the data, an indication of how much data there is, and the sort of redundancy checks you anticipate making.

By the time you have done that you should have almost answered your own question unless you enter the realms of normalising nasty data (see Ovid's comment) or you end us with a huge amount of data.

A hash would be the natural data type to store data in that is supposed to have unique keys