while( <> ){
    print "repeated address $_" if $address{$_}++;
}
[download]

To get past the flaw mentionned you can replace the appropriate line in the above code by the one below should make it work:

while (<WORDLIST>){push (@words => lc($_))}
[download]

#!/usr/bin/perl -w
use strict;

my @words;
my %list;

unless ($ARGV[1]){
    print "Usage is oneword infile outfile\n";
    exit;
}
open (WORDLIST, "$ARGV[0]")||die "Could not open file $!";
open (OUTFILE, ">$ARGV[1]")||die "Could not open file $!";
while (<WORDLIST>){push (@words => $_)}
foreach (@words){$list{$_}=$list{$_}} #this works because if the hash 
+key
                                      #exists already it is replaced!
foreach (keys %list){print OUTFILE;}
[download]

Hey slycer. You're looping around your data far too often.

while ( <WORDLIST> ) {
  print OUTFILE unless $list{lc($_)}++;
}
[download]

will be up to three time faster as it only needs to go through the data once.

(Whoops! This is essentially what IO said.)

Yah, I wrote that probably about 6 months ago as kind of a learning excersise.. I knew as soon as I posted it that it could be better, but oh well. Thanks for pointing out the errors :-)

A quick test revealed a flaw here:
what if we had :

someone@somewhere.com
and
someOne@SOMEWHERE.COM
[download]

and this is the main reason for checking for duplicate in the first place I guess..

---

Chady | http://chady.net/

That's certainly One Way To Do It, but it's certainly not the fastest.

The process goes something like this:

• read a line from the file
• stick it in the array
• repeat the previous steps for the second file
• loop through each line of the first array
• check against *every* line of the second array, with a case insensitive match
As either array grows, the number of necessary checks grows. With 2 lines in each file, you'll do four checks. With 10 lines in each file, you'll do 100 checks. (At least, if my math unit is working today.)

With the hash solution, you only loop through each file once. You don't have to check each element in one file against each element of the other file -- if it already exists in the hash, no problem. Besides that, you only have to run lc() on each line once, instead of having to build a case-insensitive version of each element in the second file for each line of the first file.

If you have 2 lines in each file, you have 4 hashings, 4 lc calls, and 4 hash assignments. No big win there. If you have 10 lines in each file, you have 20 hash assignments. You do the math for 100 elements in each file.

Besides that, it's less work. I'd say the hash is the clear winner in this case, and hopefully more people will understand why. (Apologies to the literati here for boring them. :)

• first reduce all email addresses to a canonical form by removing trailing blanks, reducing characters to lowercase, etc.
• then sort using the default lexical sort
• finally loop through the resulting list once, seeing whether any two addresses in a row are the same. If they are then delete the first and move on.

"There's more than one way to do it"

open( FH, "filename" ) || die "no way man\n";
while ( <FH> ) {
    if ( $done{$_} ne "" ) {
        $dup++;
    } else {
        $done{$_} = 1;
        push( @dups, $_ );
    }
}
[download]