remove duplicate lines from input file

heh.. it's quite a golf approach.. but it works.
(haven't tested it on a huge file .. i'm too lazy 4 now)
couldn't this be done better (more efficiently)?

perl -e 'push(@_,$_)while<>;foreach(@_){if(!exists$l{$_}){print;$l{$_}
+=0}}' <inputfile
[download]

Comment on remove duplicate lines from input file Download Code

Replies are listed 'Best First'.
RE (tilly) 1: remove duplicate lines from input file by tilly (Archbishop) on Sep 08, 2000 at 15:12 UTC
How about: `perl -ne 'print if 1 == ++$seen{$_}' file` [download] Incidentally I was involved in an interesting discussion of this topic not long ago. Some interesting material in there, including internal design philosophy differences between Python and Perl, how to handle very large files, and the fact that Java won. :-(	[reply] [d/l]
RE: RE (tilly) 1: remove duplicate lines from input file by Anonymous Monk on Sep 23, 2000 at 07:47 UTC
or: perl -ne '$seen{$_}++ or print' file	[reply]
RE: RE: RE (tilly) 1: remove duplicate lines from input file by japhy (Canon) on Sep 23, 2000 at 07:55 UTC
Or even (golf alert): `perl -ne '${$_}\|\|=print' file` (although it's not strict compliant, and breaks if strings have NULLs in them). `$_="goto+F.print+chop;\n=yhpaj";F1:eval`	[reply]
RE (tilly) 3: remove duplicate lines from input file by tilly (Archbishop) on Sep 23, 2000 at 11:27 UTC
If you follow the thread I linked you would find discussion of essentially that alternative. Using ++$seen{$_} was noticably faster because you avoid unnecessary scalar creation.	[reply]