Recommendations on a module for web crawling

edimusrex has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to crawl a web page where the data I am seeking is generated by javascript. I haven't had to deal with this before but I can imagine I am not the only one who has come across this. Is there a specific module that is preferred in this instance? I've tried using Mechanize::Firefox but the page complains about unsupported browser. I'd prefer to do this in perl but will use other languages if need be.

Thanks for your recommendations

Comment on Recommendations on a module for web crawling

Replies are listed 'Best First'.
Re: Recommendations on a module for web crawling by hippo (Archbishop) on Jul 25, 2016 at 18:31 UTC
I've tried using Mechanize::Firefox but the page complains about unsupported browser. Assuming you mean WWW::Mechanize::Firefox, then yes that would be the first port of call. If the site kicks you out perhaps they don't want any bots at all - check their Ts and Cs and honour them. Without any specifics there's little more to suggest.	[reply]
Re^2: Recommendations on a module for web crawling by edimusrex (Monk) on Jul 25, 2016 at 18:36 UTC
Thanks for the response. I am just trying to monitor a site for when it updates its inventory. The javascript must be building the list from a database before displaying it. I will look into WWW::Mechanize::Firefox some more and see if I can't dig a little deeper.	[reply]