in reply to Duplicate removal

I'd use a hash if the file isn't too big. I'd use a database or a DBM file is the file is big.

I wonder, what have you tried so far?