perlisfun has asked for the wisdom of the Perl Monks concerning the following question:

For past two years, i always use lynx as my crawling tool to collect Web Info. Recently, i found the limitation of lynx which can not crawl on any java or javascript enabled web page. I am not sure LWP can do or not. Can someone shed some light to use other perl tools to crawl on web? Thanks,

Replies are listed 'Best First'.
Re: Web Crawling automatically
by broquaint (Abbot) on May 22, 2002 at 14:54 UTC
    You're fairly spoiled when it comes to web-interaction and perl thanks to the LWP suite of modules. If you'd like info on how to use LWP check out the LWP docs and for examples see the lwpcook manpage. If you're doing simple page retrieval check out LWP::Simple which Dog and Pony recently wrote a very informative node on. If it can't be done with LWP then it's not worth doing ;-)
    HTH

    _________
    broquaint

[reply]

Back to Seekers of Perl Wisdom