Hi,

I've got a list of words (~60k words) and a big text file (~40M lines).

I would like to find the number of occurrences of each word from the list in the text file.

So far I've tried to put the list in a hash, go through each line of the text and increment the associated value each time I see it in the text (with an really basic regex).

Problem is, my code seems to be running fine on a small sample, but is taking ages with the real files. How can I improve my code ?

use strict;
use warnings;
use utf8::all;

my $listfile = "list.txt";
open IN, '<', $listfile or die "could not open $listfile";
binmode(IN, ":utf8");

open OUT, '>', "result.txt" or die "could not open out";
binmode(OUT, ":utf8");     
      
my %count;
my @lists = ();
while (my $line1 = <IN>) {
    chomp (my $key1 = (split /\t/, $line1)[0]);
    push(@lists,$key1);
}
close IN;

foreach my $list (@lists) {
    $count{$list} = 0;
}

my $textfile = "textfile.txt";
open my $fh, '<', $textfile or die "Could not open '$textfile' $!";


while (my $line = <$fh>) {
    chomp $line;
    foreach my $mot (keys (%count)) {
    chomp $mot;        
    foreach my $str ($line =~ /$mot/g) {
        $count{$str}++;
    }
    }    
}
foreach my $word (reverse sort { $count{$a} <=> $count{$b} } keys %cou
+nt) {
print OUT "$word\t$count{$word}\n";
}

[download]

In reply to Count number of occurrences of a list of words in a file by Azaghal

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.