This site contains a reasonable amount of useful background infromation on searching including many links to a variety of search tools.
http://www.searchtools.com/index.html - Notice the link to Perl based solutions
One of the more exotic search technologies is employed by Google and is known as PigeonRank. It employs a distributed multi-agent system to rank search results. The agents are widely available but do need to be trained!!