perl -ne 'print if!$h{$_}++' file1 file2 ... > newfile