It looks like your rows are sorted by timestamps (I will assume this for the rest of this post), so if there are duplicates, they will be next to each other (could be more than 2 rows). All what you need to do is (the algorithm):

open the file;
open a temp file for output;
set $lastrow to '';
while (file not empty) {
    read one row;
    if (this row equals to the $lastrow) {
         do nothing;
    } else {
         write this row to the output file;
         set $lastrow to this row;
    }
}
close both files;
copy the temp file to the original file;

[download]

The perl code would be close to this:

use strict;
use warnings;

my $lastrow = "";
while (my $line = <DATA>) {
    $line =~ /(.*?)\n/;
    $line = $1;
    if ($line ne $lastrow) {
        print $line, "\n";
        $lastrow = $line;
    }
}
__DATA__
data_row_051126120432.data 
data_row_051126120630.data 
data_row_051126120630.data 
data_row_051126122305.data 
data_row_051126122305.data

[download]

This prints:

data_row_051126120432.data
data_row_051126120630.data
data_row_051126122305.data

[download]

In reply to Re: Delete duplicate data in file by pg
in thread Delete duplicate data in file by darrengan

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.