If purchance it is necessary for you to keep the unique lines of your file in the same order, then this will remove all but the first occurance of each line and leave the remaining ones in their original order.

Just redirect the output to a new file on the command line (and uncomment the open line). 

#! perl -sw
use strict;
my %lines;
#open DATA, $ARGV[0] or die "Couldn't open $ARGV[0]: $!\n";
while (<DATA>) {
    print if not $lines{$_}++;
}

__DATA__
this is a line
this is another line
yet another
and yet another still
this is a line
more
and more
and even more
this is a line
and this
and that
but not the other cos its a family website:)

[download]

Gives 

C:\test>uniq
this is a line
this is another line
yet another
and yet another still
more
and more
and even more
and this
and that
but not the other cos its a family website:)

C:\test>

[download]

The caveat of course is that with a large file, that hash could get mind of big, but maybe that's ok if this is what you need to do.

Cor! Like yer ring! ... HALO dammit! ... 'Ave it yer way! Hal-lo, Mister la-de-da. ... Like yer ring!

In reply to Re: Remove Duplicate Lines by BrowserUk
in thread Remove Duplicate Lines by dcb0127

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.