A more normal way to test for duplicates is to use a hash. You can set the input record seperator to @: to make this easy. There are a couple of ways to go about this depending on if order is important to you, if you need to know when a duplicate was removed and if the entire file can be read into memory at one time.

# here is an example to get you started in untested code.

my ($in, $out, %hash);
open $in, "<", $file_name; # you set the name somewhere else
open $out, ">", $out_file;
local $/ = "@:";           # set the input record seperator
while (<$fh>) {
    if ($hash{$_}++) {     # will be undef first time then >0
        print "Warning duplicate found\n$_";
    } else {
        print $out $_
    }
}
[download]

If the file is too big to hold in the hash then perhaps use the first line of each record as a key and store the ofset in the outfile where it can be found. If you find that key again read the one you wrote back to see if the entire record matches. Of course the value under the key would be an array of ofsets to allow multiple differing records with the same first line.

If you control the writing program perhaps you can better fix that not to write duplicte records.

You would also get more out of the Monastery if you had a quick look here How do I post a question effectively?

Cheers,
R.