in reply to Own robot to search engine

In addition to what robot_tourist has already advised, you could have a read at Designing a Search Engine, which raises a couple of things you may want to think about. You seem to have an idea about how you want this to work. With this in mind read Re^2: WWW::Mechanize problem, here someone has a list of URLs they wish to visit/parse. Limbic~Region provides a detailed description of the how to deal with these situations, which may be of interest to you also.

Update: used Super Search and added a link to a previous question related to this topic.

Hope this helps

Martin