that algo limits the scope of regex matches to individual terms.

Allowing for post-wildcarded keywords stem* is relatively easy, especially if you limit the prefix to a minimum of (say) 3-chars or more.

Where I had to provide for a pre & post wildcards, I built a trigram index (and insisted upon a minimum of 3 contiguous chars). That worked well, but required a substantial sized index.

The one search engine I am familiar with that does allow full regex including term spanning is Google's codesearch. I'd love to see insights into how they do that. It's quite interesting to see the difference in the time it takes to do

Really quite remarkable even given the massive parallelism. They must do some pretty clever parsing of the regexes in order to avoid searching the entire dictionary. I guess the code corpus is considerably smaller than the main set which will help a lot.

That said: It makes you wonder why the OP doesn't just use Googles search engine?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP PCW It is as I've been saying!(Audio until 20090817)

In reply to Re^3: What DB style to use with search engine by BrowserUk
in thread What DB style to use with search engine by halfcountplus

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.