in reply to Fulltext DB search: The Need for Speed

In addition to Abigail's excellent advice, heuristics can be employed to speed things up.

One heuristic is to create an internal stoplist of common words and look up only non-stoplisted words first. Then from that small set of matching records, search for the stoplisted words. Set intersection is associative -- use that to your advantage.

Another heuristic is to use caching. If there are a relatively small number of very common and commonly requested words, create a table of matching records for each word and search only that table if the word is in the phrase. This prevents having to pull up most of a table each time such a word appears.

-Mark

  • Comment on Re: Fulltext DB search: The Need for Speed