punkish has asked for the wisdom of the Perl Monks concerning the following question:

There probably is already a CPAN module for this, but I am stuck trying to figure out a solution. Given a search term in a paragraph of text, I want to extract n words on either side of the match. I am sure modules such as Plucene or KinoSearch already do something like this, but I am curious about the mechanics of performing such a task. Don't give me the code, but please point me toward how such a problem would be solved.

For example, if my search term was 'match' and 'n' 10, given the above para, I would get the following string

...want to extract n words on either side of the match. I am sure modules such as Plucene or KinoSearch already...

--

when small people start casting long shadows, it is time to go to bed

Replies are listed 'Best First'.
Re: extracting search term context
by BrowserUk (Patriarch) on Jun 09, 2010 at 03:48 UTC

    It's harder to describe without code than with.

    First define what constitutes a word. Upper and lower case letters obviously, but what about the quote in "don't"?

    Then define what comes in between 'words'. Spaces for sure, but what about punctuation like ,.?!- etc.

    Once you've defined your words and separators, you want to capture n words and separators before the match and n separators and words after.

    $word = qr[[a-zA-Z']+];; $sep = qr[[ \t.,!?]+];; $n=10; $match = 'match'; $re = qr[((?:$word$sep){$n}\Q$match\E(?:$sep$word){$n})];; print $para =~ $re;; want to extract n words on either side of the match. I am sure modules + such as Plucene or KinoSearch already

    So far, so good, but what happens if the word is less that n from the start or end? It won't match, so the quantifier has to become {0,$n}:

    $re = qr[((?:$word$sep){0,$n}\Q$match\E(?:$sep$word){0,$n})];; $match = 'CPAN';; print $para =~ m[$re];; There probably is already a CPAN module for this, but I am stuck tryin +g to figure

    Better, but it will still fail if the definition of words and separators isn't good enough

    For example, if there are numbers in there somewhere. Or brackets. Or quotes. And that's before you get to trying to deal with unicode.

    At best the above is a starting point. How far you need to go beyond that will depend upon your application.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      probably a little better would be to use \b and \w instead of defining a word and separator from sratch, or even use the posix equivalents for more clarity. also a further suggestion to the original question, an alternative would be to read the file in the desired chunks/units to start with, each time through the loop reading one extra word from the file into this comparison buffer and dropping one word off at the end of the buffer. In other words a sliding window/buffer technique with word being step size.
      the hardest line to type correctly is: stty erase ^H
        probably a little better would be to use \b and \w instead of defining a word and separator from sratch

        I invite you to demonstrate this.

        reading one extra word from the file

        How do you read a "word" from a file?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: extracting search term context
by Anonymous Monk on Jun 09, 2010 at 04:09 UTC
    Plucene and KinoSearch create indexes (giant hash), so if you lookup match , in return you get a list of files which contain match.

    Then you iterate over the list of files and call extract($file,'match',10)

    extract() then uses sliding window technique to perform a linear search, that is, you read the file word by word (or line by line, and then split into words), while keeping a buffer of the previous N words until you find your match, then you print N words from the previous buffer, and then read /print additional N words.

      Ideally you would cache these search results, perhaps invalidating the cache when you add/index more files.
Re: extracting search term context
by bluescreen (Friar) on Jun 09, 2010 at 03:32 UTC

    A regular expression is all what you need, take a look at perlre. Just to give you an example see if the following code does what you want:

    my $text = 'whatever you want in here'; my $n = 10; my $word = 'match'; if ( $text=~ /(([.\s]+\w+){$n}[.\s]+$word[.\s]+(\w+[.\s]+){$n})/ ) { print $1; }