If you can depend on the datestamps then pg's solution is fine. I like the hash solution better myself for small to medium data files.

Example:

my %seen = ();
while(<>) {
     print if ! $seen{$_};
     $seen{$_} = 1;
}
[download]

Then process your file like this:
./above_code.pl < data_file_in > data_file_out

You might want to chomp the data lines so you don't miss trailing new lines. Also if the data files are huge, then go with pg's solution, this one will create hash keys for each data line in the file.

Steve