Scraping javascript results from webpages

ghawkins has asked for the wisdom of the Perl Monks concerning the following question:

I've asked around quite a bit already, thought I'd try here. We want to pull the contents of a large number of webpages under test and verify the contents of a section of javascript post-rendering. I'm a big fan of LWP, however, it will only return the html source. I imagine this has been asked before ... Is there a module or even some other approach that will work? Thanks, Gary

Comment on Scraping javascript results from webpages

Replies are listed 'Best First'.
Re: Scraping javascript results from webpages by Joost (Canon) on Dec 02, 2006 at 00:10 UTC
If you haven't already, take a look at Win32::IE::Mechanize or Mozilla::Mechanize. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]
Re: Scraping javascript results from webpages by perrin (Chancellor) on Dec 01, 2006 at 20:04 UTC
Sure, you can do that with Selenium. However, if you are okay with testing the server-side code and don't need to specifically test the javascript, LWP is fine.	[reply]
Re^2: Scraping javascript results from webpages by ghawkins (Initiate) on Dec 01, 2006 at 23:15 UTC
"post-rendering" meaning the results/output of the javascript, like the user sees it, not the page source.	[reply]
Re^3: Scraping javascript results from webpages by perrin (Chancellor) on Dec 02, 2006 at 05:55 UTC
Sure, but that only matters if you're trying to test the JavaScript itself. Otherwise, it's irrelavant.	[reply]