scrape site after javascript

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Is it possible to scrape a site AFTER the JavaScript has been rendered?

There are some sites that have links to press depending on what browser you are using. Since this is JavaScript, my Lwp::Simple or WWW::Mechanize can't get past these links.

Is there any way to do this?

Comment on scrape site after javascript

Replies are listed 'Best First'.
Re: scrape site after javascript by davidrw (Prior) on Mar 25, 2006 at 03:03 UTC
After it's been rendered by _what_? Obviously your LWP::Simple/WWW::Mechanize (both based on LWP) can't scrape it post-javascript cause they can't process the javascript ... You would have to have some other browser in the middle ... for example you might be able to use Win32::IE::Mechanize to get the rendered html .. or if that doesn't work maybe there's a way w/Win32::OLE and drilling through the document's object model .. Don't know how, but hopefully it'll point you in the right direction (assuming you're on windows of course)...	[reply]
Re: scrape site after javascript by perrin (Chancellor) on Mar 25, 2006 at 04:54 UTC
Usually the JavaScript just makes pretty simple changes which you can emulate yourself in Perl.	[reply]