in reply to Re^6: crawling one website
in thread crawling one website

By real links I mean full kinks started with http://... not links to sub-directories.
The program you recommended seems to be what I need. It looks like it retrieves all "real" links from a webpage, but it does not go over a domain tree. So, in order to get all links starting from the root I may use some program (say WWW::Sitemap) which retrieves urls of all depth levels and inside each one I can use hgrepurl.pl to get all links from there.
Am I right?

Replies are listed 'Best First'.
Re^8: crawling one website
by planetscape (Chancellor) on May 29, 2011 at 03:10 UTC