Take a look at mnoGoSearch, they have done all the work for you already. I've got very good results with it.

From their site: "mnoGoSearch (formerly known as UdmSearch) is a full-featured web search engine software for intranet and internet servers. mnoGoSearch for UNIX is a free software covered by the GNU General Public License"

It's C based code, and the compiled CGI frontend works far better than the provided Perl script. Maybe some day someone will write an XS interface...