Anfield has asked for the wisdom of the Perl Monks concerning the following question:

There was an LWP spider script on perl website like 2 years ago. I can not find it any more now. Does any one here have a similiar script? The script needed must be able to go deep at a pre-set value to fetch links. thanx in advance

Replies are listed 'Best First'.
Re: LWP spider
by vagnerr (Prior) on Mar 20, 2002 at 19:58 UTC
    When you install LWP a couple of perl scripts are installed at the same time called lwp-rget and lwp-request. lwp-rget is quite a nice spider program supporting basic auth and cookie files so that may be what you're looking for.

    ---If it doesn't fit use a bigger hammer
Re: LWP spider
by tachyon (Chancellor) on Mar 20, 2002 at 20:43 UTC

    Link Checker is an LWP NET::FTP HTML::TokeParser link checking spider I wrote that crawls a site width first. merlyn also adds links to spiders of his at this node (no less than four different ones!). Take your pick.

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print