When you want to avoid duplicates, the automatic answer is 'use a hash'.
Simply put some unique value (or the whole line) as a key for a hash. I'll let you decide what is an appropriate value for this key - it should be easily generated, and unique for each identical record.
Do a: $hash{$key}++;
when you find a value you want for the first time.
Then, skip that line when you encounter the same key again. Put: next if(exists $hash{$key});
at the top of your loop.