in reply to removing duplicate lines
perl -ni -e 'print if $_ ne $old;$old = $_' duplicates [download]