in reply to Detecting duplicate entries

Just in case you didn't know, under Unix :

cat file.txt | sort | uniq
will give you the content of your file file.txt with all duplicates deleted (and sorted by the way...)

UPDATE : Once again I was too long typing, ignore this post as zigster gave this answer faster than me...