full text search (with lots of text)

glwtta has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks,

So I have records consisting of an id and a chunk of text (a medium sized paragraph) and I want to search them for specific keywords, and more importantly exact phrases. Simple enough, except that I have about 13 million records, taking up about 15 gigs.

Right now I have this indexed using MySQL 4, it does very well for keyword searches, but is far too slow on the exact phrases.

Any suggestions on other tools to do this? It takes about a minute and a half to grep through that file - surely one can do better? :)

Comment on full text search (with lots of text)

Replies are listed 'Best First'.
Re: full text search (with lots of text) by simonm (Vicar) on Jul 21, 2003 at 22:07 UTC
I've done this in the past with a compromise technique: using the MySQL keyword search on the words contained in the phrases, and then looping over those records in Perl, checking for full-phrase matches.	[reply]