If on *nix you can pipe the sort output into uniq (http://en.wikipedia.org/wiki/Uniq) to get rid of adjacent duplicates.

knoppix@Microknoppix:~$ cat rubbish
cat
fish
dog
apple
cat
bird
knoppix@Microknoppix:~$ sort rubbish | uniq
apple
bird
cat
dog
fish
knoppix@Microknoppix:~$
[download]

I hope this is of interest.