my %seen;

while (<>) {
  next if $seen{$_};
  print;
  $seen{$_}++;
}
[download]

To avoid redundancy in your code, you could do this:

my %seen;

while (<>)
{
     next if ($seen{$_}++);
     print;
}
[download]

You can do it in one shot, so you might as well. Note that this code eliminates all duplicate lines, not just repeated ones. If you want to just ditch repeats, use this:

my $last;

while (<>)
{
     next if ($_ eq $last);
     $last = $_;
     print;
}
[download]

Thus lines "A A A B B B A A C C" will be "A B A C" not "A B C" as in the previous bit.

You can do it in one shot, so you might as well.

That makes your second snippet

my $prev;

while (<>) {
    next if ($_ eq $prev);
    print $prev = $_;
}
[download]

:^)

Wait, we can shorten that..

my $prev;

while (<>) {
    print $prev = $_ unless $_ eq $prev;
}
[download]

Hmm..

my $prev;
$_ ne $prev and print $prev = $_ while <>;
[download]

Err.. sorry, got carried away for a second.. Perl is just too seductive. Sigh. :-)

Makeshifts last the longest.

This won't do exactly as the AM wants - some lines will be duplicate to the user, but not to Perl:

$ more file.txt
N AB TX NC
AB N TX NC
FOO BAR
N AB TX NC
$ perl test.pl file.txt
N AB TX NC
AB N TX NC
FOO BAR
[download]

The first two lines of the file.txt file are "the same" to the user, but not to your program. zejames' solution works to the AM's needs, as it creates an unique key for the hash, based on the AM's definition of a duplicate.

Jason

# We are modifying the $/ variable, so we limit the scope
# by adding some {} around the code
{
  local $/ = '';
  $^I      = '.bak';  # See man perl and the -i switch for that trick
  @ARGV    = ('data.txt');
  while (<>) {
    # The order is not important, so we sort the fields to
    # obtain a unique id
    $sorted = join ':', sort split /\s+/; 
    print if (! $seen{$sorted}++ );
  }
}
[download]

my %hash
while(<>){
   my $key = join " ", (sort (split " "));
   $hash{$key} = 1;
}
#now iterate over the keys of the hash, and either print them out, or 
+do your insert in to the database
[download]

thor