Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks

I want to save some of my time of processing so I want to do the duplicated line detection (and compression). The duplication definition is simple: If a line is same as its previous line, it will be consider as duplicate...any one can provide a fast solution?

Replies are listed 'Best First'.
Re: Detect Duplicated line
by ikegami (Patriarch) on Apr 05, 2008 at 01:04 UTC
    A simple check in your read loop will handle that: 
    my $last_line;
while (<$fh>) {
   chomp;

   # Skip duplicates
   next if $.>1 && $_ eq $last_line;
   $last_line = $_;

   ...proceed as normal...
}

    [download]
[reply]
[d/l]
Re: Detect Duplicated line
by apl (Monsignor) on Apr 05, 2008 at 12:00 UTC
    If the file exists before your Perl script starts (and you're on *nix), you could uniq - d the file first.
[reply]

Back to Seekers of Perl Wisdom