in reply to Own robot to search engine

What do you need more help with? The perl or the design?

I'm guessing you could use LWP let your script access the web, but I have no experience with it. For the design side, I'm just throwing an untested idea into the air: give your script a starting address, collect all the links on that page and go to each one link turn; and for each of those links, follow all the links on the page and so on. There may be robot etiquette you should be aware of, see this page (near the top of a google search for 'search robot etiquette'), which has lots of other search robot related information.

How can you feel when you're made of steel? I am made of steel. I am the Robot Tourist.
Robot Tourist, by Ten Benson