in reply to removing non-duplicates

Apart from simply adding the -u option to unix sort, here are two perl options:

1) preserving the existing order:

perl -e 'while(<>){ $_{$_} or print $_; $_{$_}=1;}' <file
2) sorted
perl -e 'while(<>){ $_{$_}=1;} print sort keys %_;' <file
Note: I deem it to be safe enough to use %_ for quick perl -e usages (without any modules) and not for normal programming where the hash %_ should be replaced by a properly named and declared one.

One world, one people