in reply to Calculating "similarity"

OK, let me rephrase your problem to see if I understood you correctly. I assume that you want to compare several (other) files to one base file. You want to calculate a similarity rating between those files based on the number of words that are same (=intersection) and the total number of words (=total) in each of the other files - this last point I'm not sure about, it could also be total number of words in base and other file together.

Does this do what you want?

#!/usr/local/bin/perl -w use strict ; my $stopfile = 'stopwords'; my $base= shift @ARGV; # read in stopwords open STOP, "<$stopfile" or die $!; chomp( my @stop= <STOP> ); close STOP; my %stopwords=(); # add the empty string '' to the stopwords as well @stopwords{@stop,''} = (); # read in basefile my %base_filterwords=(); open BASETEXT, "<$base" or die $!; while ( <BASETEXT> ) { @base_filterwords{ map { my $l = lc; exists $stopwords{$l}?():$l } +split /\W+/ } = (); } close BASETEXT; # read in other files my %other_filterwords=(); while ( <> ) { @other_filterwords{ map { my $l = lc; exists $stopwords{$l}?():$l } + split /\W+/ } = (); } continue { if (eof) { my $total = scalar keys %other_filterwords; my $intersect = 0; for my $key (keys %other_filterwords) { $intersect++ if exists $base_filterwords{$key}; } my $sim = 2*$intersect/$total; print "Similarity between $base and $ARGV = $sim\n"; print "\t@{[keys %other_filterwords]}\n"; print "\t@{[keys %base_filterwords]}\n"; # reset other filter words %other_filterwords = (); } }

Update Fixed some small bugs in the code ...

-- Hofmator