in reply to Creating a web crawler (theory)
To grab pages, you already know about LWP::UserAgent. To extract links, you can use HTML::SimpleLinkExtor, which base turn relative URLs into absolute ones, or if you need something more fancy, you can write your own subcless of HTML::Parser.
If you already have the URLs and you want to turn relative URLs into absolute ones, URI can do that for you.
You can look at my personal web snarfer, webreaper, which has code fora lot of the things you need to do. Steal what you need.
|
|---|