in reply to remove duplicate lines from input file

How about: 
perl -ne 'print if 1 == ++$seen{$_}' file

[download]
Incidentally I was involved in an interesting discussion of this topic not long ago. Some interesting material in there, including internal design philosophy differences between Python and Perl, how to handle very large files, and the fact that Java won. :-(

Replies are listed 'Best First'.
RE: RE (tilly) 1: remove duplicate lines from input file
by Anonymous Monk on Sep 23, 2000 at 07:47 UTC
    or: perl -ne '$seen{$_}++ or print' file
[reply]
      Or even (golf alert): perl -ne '${$_}||=print' file (although it's not strict compliant, and breaks if strings have NULLs in them).

      $_="goto+F.print+chop;\n=yhpaj";F1:eval
[reply]
      If you follow the thread I linked you would find discussion of essentially that alternative. Using ++$seen{$_} was noticably faster because you avoid unnecessary scalar creation.
[reply]

In Section Cool Uses for Perl