LWP spider

Anfield has asked for the wisdom of the Perl Monks concerning the following question:

There was an LWP spider script on perl website like 2 years ago. I can not find it any more now. Does any one here have a similiar script? The script needed must be able to go deep at a pre-set value to fetch links. thanx in advance

Comment on LWP spider

Replies are listed 'Best First'.
Re: LWP spider by vagnerr (Prior) on Mar 20, 2002 at 19:58 UTC
When you install LWP a couple of perl scripts are installed at the same time called lwp-rget and lwp-request. lwp-rget is quite a nice spider program supporting basic auth and cookie files so that may be what you're looking for. ---If it doesn't fit use a bigger hammer	[reply]
Re: LWP spider by tachyon (Chancellor) on Mar 20, 2002 at 20:43 UTC
Link Checker is an LWP NET::FTP HTML::TokeParser link checking spider I wrote that crawls a site width first. merlyn also adds links to spiders of his at this node (no less than four different ones!). Take your pick. cheers tachyon s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print	[reply]