How are you processing the list so that 100 addresses take 1 seconds to find duplicates? If you use a hash, Perl should find duplicates in 100 lines in milliseconds. With the whole file, you might have to worry about memory usage. Without swapping, the processing should be very fast.

my %seen;
while (my $line = <$fh>) {
    chomp($line);
    print "$line\n" unless seen{$line};
    $seen{$line}++;
}
[download]