in reply to Detecting duplicate entries

Just in case you didn't know, under Unix :

cat file.txt | sort | uniq
will give you the content of your file file.txt with all duplicates deleted (and sorted by the way...)

UPDATE : Once again I was too long typing, ignore this post as zigster gave this answer faster than me...

Replies are listed 'Best First'.
Re: Re: Detecting duplicate entries
by merlyn (Sage) on Mar 01, 2001 at 22:08 UTC
      And to do it all at once as an inline edit:
      sort -uo file.txt file.txt

      Update: For those that asked, sort -u file.txt > file.txt
      will completely and happily change file.txt into a zero byte file, hence the need for the "o" flag.

Re: Re: Detecting duplicate entries
by arhuman (Vicar) on Mar 01, 2001 at 22:15 UTC
    ouch ! Didn't know this sort switch !

    (Oh my god, I just realized how many useless char I typed during all those years...)