in reply to need help determining which web browsing module to use

I am certainly no expert on HTML. I am not sure exactly what you have. One line doesn't tell me much. Over the years I've written a few web scrapers with LWP and a couple with WWW::Mechanize. As long as the webpage is serving up just HTML code instead of javascript, you can use the base WWW::Mechanize module. That's been the case so far in my current applications. If the webpage requires executing javascript code, then Perl cannot do that alone. In that case, you will need WWW:Mechanize::Chrome or similar. In that case, Perl controls the browser and has the browser execute the Javascript code. Mechanize sees the result of what the browser's javascript code did.

I would start by reading Cpan Mech Docs and then take a look at some Mech examples. Then I would start "hacking" and experimenting and see how far you can get with the base Mech module. If you are using a public, heavily trafficked web site, then show us the URL.

Also be aware of the potential impact that your code could have on the target web site. I have one application that "beats up" one web site pretty good. But I have agreement with the site owner about what hours and what days my application can run. This is an important consideration if you are going to retrieve a lot of data.

Update: s/Java/Javascript/; #Completely different things!

  • Comment on Re: need help determining which web browsing module to use

Replies are listed 'Best First'.
Re^2: need help determining which web browsing module to use
by marto (Cardinal) on Nov 14, 2020 at 20:44 UTC

    'Perl controls the browser and has the browser execute the Java code'

    JavaScript, not Java.

      Thanks for the correction!