i was interested in writing my own little search engine. Thought it would not be interesting to make it a simple grep-like thing. I implemented the proximity search algorithm described @ http://citeseer.nj.nec.com/cachedpage/457664/1

please note that this is code is nowhere near perfect.

use warnings; use strict; { my $text = "this is the text to search"; my $index = parseText($text); print getScore($index, ['this', 'search']) ."\n"; # this prints about ~17, terms are 4 words appart } { my $text = "and this is some more text"; my $index = parseText($text); print getScore($index, ['this', 'text']) ."\n"; # this prints about ~23, terms are only 3 words appart } sub getScore { my ($index, $words) = @_; my ($score, $num_words); foreach my $word (@$words) { unless ($index->{$word}) { return -1 } $score += 1 + (1 - 1/scalar @{ $index->{$word} }); $num_words++; } return $score if $num_words <= 1; my $proximities = getMinProx_PS($index, $words); my $prox_count = 1/1.5; my $prox_bonus = 0; foreach my $prox (@$proximities) { $prox_count *= 1.5; $prox_bonus = (100/(($prox->[1] - $prox->[0] + 1)/$num_words)* +*1.7)/$prox_count**2; last if $prox_bonus < 0.01; $score += $prox_bonus; } return $score; } sub getMinProx_PS { my ($index, $keywords) = @_; my (@cur_list, @minimal); foreach my $keyword (@$keywords) { return undef unless ref $index->{$keyword}; push @cur_list, [shift @{ $index->{$keyword} }, $keyword]; } @cur_list = sort {$a->[0] <=> $b->[0]} @cur_list; while (1) { if (scalar @{$index->{ $cur_list[0][1] }} == 0) { push @minimal, [$cur_list[0][0], $cur_list[-1][0]]; last; } my $p = [shift @{$index->{ $cur_list[0][1] }}, $cur_list[0][1] +]; my $q = [$cur_list[1][0], $cur_list[1][1]]; if ($p->[0] > $cur_list[-1][0]) { push @minimal, [$cur_list[0][0], $cur_list[-1][0]]; shift @cur_list; push @cur_list, $p; } else { $cur_list[0] = $p; @cur_list = sort {$a->[0] <=> $b->[0]} @cur_list; } } @minimal = sort {$a->[1] - $a->[0] <=> $b->[1] - $b->[0]} @minimal +; return \@minimal; } sub parseText { my $text = shift; my (%word_index, $word_count); while ($text =~ m/(\w+)/g) { push @{$word_index{lc $1}}, ++$word_count; } return \%word_index; }

In reply to Proximity Search by ClickToViewInfo

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.