Switching from using a hash to a bit vector to perform the intersection saves another 40%; so 5 seconds rather than 25:

#! perl -slw use strict; use Time::HiRes qw[ time ]; sub MI { my( $string_es, $string_en, $hash_es, $hash_en ) = @_; my $n_array_es = $hash_es->{ $string_es } =~ tr[ ][ ]; ++$n_array_es; my $n_array_en = $hash_en->{ $string_en } =~ tr[ ][ ]; ++$n_array_en; my $prob_es = ( $n_array_es ) / 6939873; my $prob_en = ( $n_array_en ) / 6939873; my $intersection = Intersection( \$hash_es->{ $string_es }, \$hash_en->{ $string_en } ); my $prob_es_en= $intersection / 6939873; $prob_es_en = ( $prob_es_en + ( $prob_es * $prob_en * 0.1 ) ) / 1. +1; my $mi = $prob_es_en * log( $prob_es_en / ( $prob_es * $prob_en) ) +; return $mi; } sub Intersection { my( $refA, $refB ) = @_; my $bits = ''; vec( $bits, $1, 1 ) = 1 while $$refA =~ m[(\S+)]g; my $intersects = 0; vec( $bits, $1, 1 ) && ++$intersects while $$refB =~ m[(\S+)]g; return $intersects; } our $N //= 1e4; my $hypo = 'fred'; my $text = 'bill'; my %hash_es; $hash_es{ $hypo } = join ' ', 1 .. $N; my %hash_en; $hash_en{ $text } = join ' ', 1 .. $N; my $start = time; my $MI_T = MI( $hypo, $text, \%hash_es, \%hash_en ); printf "Took: %f seconds\n", time() - $start; __END__ C:\test>888162 -N=3e6 Took: 5.966000 seconds

Any further savings would probably need to come from where you build the lists of line numbers. Ie. If you built up a bit vector for each keyword where a set bit indicated that keyword was found in that line rather than concatenating the line numbers into a string. You can the count the populations using

my $n_es = unpack '%32b*', $hash_es->{ $string_es }; my $n_en = unpack '%32b*', $hash_en->{ $string_en };

And perform the intersection using:

my $intersections = unpack '%32b*', $hash_es->{ $string_es } & $hash_es->{ $string_es };

Both of which are extremely fast operations. If you put that all together, then you reduce the 25 seconds to 2 milliseconds, which is roughly 125,000 times faster:

#! perl -slw use strict; use Time::HiRes qw[ time ]; sub MI { my( $string_es, $string_en, $hash_es, $hash_en ) = @_; my $n_es = unpack '%32b*', $hash_es->{ $string_es }; my $n_en = unpack '%32b*', $hash_en->{ $string_en }; my $prob_es = ( $n_es ) / 6939873; my $prob_en = ( $n_en ) / 6939873; my $intersection = unpack '%32b*', $hash_es->{ $string_es } & $hash_es->{ $string_es }; my $prob_es_en= $intersection / 6939873; $prob_es_en = ( $prob_es_en + ( $prob_es * $prob_en * 0.1 ) ) / 1. +1; my $mi = $prob_es_en * log( $prob_es_en / ( $prob_es * $prob_en) ) +; return $mi; } our $N //= 1e4; my $hypo = 'fred'; my $text = 'bill'; my %hash_es; $hash_es{ $hypo } = ''; vec( $hash_es{ $hypo }, $_, 1 ) = 1 for 1 .. $N; my %hash_en; $hash_en{ $text } = ''; vec( $hash_en{ $text }, $_, 1 ) = 1 for 1 .. $N; my $start = time; my $MI_T = MI( $hypo, $text, \%hash_es, \%hash_en ); printf "Took: %f seconds\n", time() - $start; __END__ C:\test>888162 -N=3e6 Took: 0.001901 seconds

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re: improving the speed by BrowserUk
in thread improving the speed by perl_lover_always

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.