Web Spidering module?

elbie has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Web Spidering module? by charnos (Friar) on Sep 23, 2001 at 02:26 UTC
While I've never used the module myself, WWW::Robot seems to be exactly what you are looking for. It seems to be a decent solution, but it has alot of external module dependencies. Good luck spidering!	[reply]
Re: Web Spidering module? by Starky (Chaplain) on Sep 23, 2001 at 10:46 UTC
I can't seem to dig up the specific reference, but there was a Perl Journal article awhile back which, if I recall correctly, had a 4 line web spider. If someone out there could dig that up, it would be worth a read if only for amusement value :-)	[reply]
Re: Web Spidering module? by jehuni (Pilgrim) on Sep 23, 2001 at 23:44 UTC
Yeah, I saw that one, too. It was actually a one-liner: A Web Spider in One Line? The article also mentions a good source of information about spiders: The Web Robots Pages. -jehuni	[reply]
(elbie): Web Spidering module? by elbie (Curate) on Sep 24, 2001 at 18:35 UTC
I found HTML::LinkExtor (also used in the one line spider mentioned above) on the weekend, and it was quite helpful. Thanks everybody! elbieelbieelbie	[reply]
Re: Web Spidering module? by jryan (Vicar) on Sep 23, 2001 at 02:22 UTC
Believe it or not, I started working on one just the other day... I'll post it when I am done (which will probably be the next chance I get a large chunk of time to code). However, if anyone knows if something like this is already in existence, please let me know so I can work on something else :)	[reply]