While writing your own search engine might well be fun I would recommend using an open source one like swish-e to do the heavy lifting generating the index. It has a nice set of Perl bindings and works very well.