in reply to removing duplicate lines

Why make things complicated. A little magic with the perl switches and you are there.
perl -ni -e 'print if $_ ne $old;$old = $_' duplicates