In addition to what robot_tourist has already advised, you could have a read at Designing a Search Engine, which raises a couple of things you may want to think about. You seem to have an idea about how you want this to work. With this in mind read Re^2: WWW::Mechanize problem, here someone has a list of URLs they wish to visit/parse. Limbic~Region provides a detailed description of the how to deal with these situations, which may be of interest to you also.
Update: used Super Search and added a link to a previous question related to this topic.
Hope this helps
Martin | [reply] |
What do you need more help with? The perl or the design?
I'm guessing you could use LWP let your script access the web, but I have no experience with it. For the design side, I'm just throwing an untested idea into the air: give your script a starting address, collect all the links on that page and go to each one link turn; and for each of those links, follow all the links on the page and so on. There may be robot etiquette you should be aware of, see this page (near the top of a google search for 'search robot etiquette'), which has lots of other search robot related information.
How can you feel when you're made of steel? I am made of steel. I am the Robot Tourist.
Robot Tourist, by Ten Benson
| [reply] |
| [reply] |
Parsing text wouldn't be the way to go; you can use google services to get search results and format them to your liking (subject to google's terms and conditions). Writing robots to crawl the entire web and a search engine to search the results is an enormously large project. It would really surprise me if it weren't better to use an existing search engine.
For those that haven't seen it, the Binary Search Tree 2 and the report on it are interesting reading. | [reply] |
| [reply] |