Hello,
I have the following perl script which compares words in the 2nd field of two csv files, and if 5 or more words match then it prints both line together as a positive match.
#!/bin/perl
+
+
my @csv2 = ();
+
open CSV2, "<csv2" or die;
+
@csv2=<CSV2>;
+
close CSV2;
+
+
my %csv2hash = ();
+
for (@csv2) {
+
chomp;
+
my ($title) = $_ =~ /^.+?,\s*([^,]+?),/; #/ match the title
+
$csv2hash{$_} = $title;
+
}
+
+
open CSV1, "<csv1" or die;
+
while (<CSV1>) {
+
chomp;
+
my ($title) = $_ =~ /^.+?,\s*([^,]+?),/; #/ match the title
+
my %words;
+
$words{$_}++ for split /\s+/, $title; #/ get words
+
## Collect unique words
+
my @titlewords = keys(%words);
+
my @new; #add exception words which shouldn
+'t be matched
foreach my $t (@titlewords){
+
push(@new, $t) if $t !~ /^(rare|vol|volume|issue|double|magazi
+ne|mag)$/i;
}
+
@titlewords = @new;
+
my $desired = 5;
+
my $matched = 0;
+
foreach my $csv2 (keys %csv2hash) {
+
my $count = 0;
+
my $value = $csv2hash{$csv2};
+
foreach my $word (@titlewords) {
+
my @matches = ( $value=~/\b$word\b/ig );
+
my $numIncsv2 = scalar(@matches);
+
@matches = ( $title=~/\b$word\b/ig );
+
my $numIncsv1 = scalar(@matches);
+
++$count if $value =~ /\b$word\b/i;
+
if ($count >= $desired || ($numIncsv1 >= $desired && $numI
+ncsv2 >= $desired)) {
$count = $desired+1;
+
last;
+
}
+
}
+
if ($count >= $desired) {
+
print "$csv2\n";
+
++$matched;
+
}
+
}
+
print "$_\n\n" if $matched;
+
}
+
close CSV1;
I would now like to add extra functionality so that- if both the lines contain a year, in the format 1989, then the years in each csv have to match for it to be considered a positive match.
However, if only one of the lines contains a year, then I would like the usual 5 matching words rule to apply and the year becomes irrelevant.
Examples just to clarify.
In this example field 2 contains 5 matching words but the two years (1973 + 2013) are different so this would be discounted as a match:
2523021356, RARE TV RADIO TIMES MAGAZINE DOCTOR WHO 1973 THE THREE 3
+DOCTORS DR JON PERTWEE, http://www.example.co.uk, 12
12278788, TV & SATELLITE WEEK 11 MAY 2013 GILLIAN ANDERSON DOCTOR WHO
+NOT RADIO TIMES , http://www.example.co.uk, 12
In this example the years are the same AND there are 5 or more matching words so this would be a positive match:
2523021356, RARE TV RADIO TIMES MAGAZINE DOCTOR WHO 1973 THE THREE 3
+DOCTORS DR JON PERTWEE, http://www.example.co.uk, 12
12278788, TV & SATELLITE WEEK 11 MAY 1973 GILLIAN ANDERSON DOCTOR WHO
+NOT RADIO TIMES , http://www.example.co.uk, 12
In this example, only one of the titles contain a year (1973), but there are also 5 or more matching words, I would like this to be considered a positive match:
2523021356, RARE TV RADIO TIMES MAGAZINE DOCTOR WHO 1973 THE THREE 3
+DOCTORS DR JON PERTWEE, http://www.example.co.uk, 12
12278788, TV & SATELLITE WEEK 11 MAY GILLIAN ANDERSON DOCTOR WHO NOT R
+ADIO TIMES , http://www.example.co.uk, 12
In this example, none of the titles contains a year, but 5 or more words match so this would be a positive match:
2523021356, RARE TV RADIO TIMES MAGAZINE DOCTOR WHO THE THREE 3 DOCTO
+RS DR JON PERTWEE, http://www.example.co.uk, 12
12278788, TV & SATELLITE WEEK 11 MAY GILLIAN ANDERSON DOCTOR WHO NOT R
+ADIO TIMES , http://www.example.co.uk, 12
How can I add this functionality to the script without making wholesale changes?
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
| |
For: |
|
Use: |
| & | | & |
| < | | < |
| > | | > |
| [ | | [ |
| ] | | ] |
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.