Take a look at Search Tools - Perl Search Tools Product Listings for inspiration. The link at the top - Roll Your Own Search Engine may be applicable. The O'Reilly website has many articles which will intrest you, just search for 'spider'.