Re^2: How to scrape an HTTPS website that has JavaScript

tomhukins,
You have succesfully gotten HTTP::Recorder to do JavaScript? You might want to let the author know, since the docs for even the latest development release state it won't record JavaScript Actions.

Cheers - L~R

Comment on Re^2: How to scrape an HTTPS website that has JavaScript

Replies are listed 'Best First'.
Re^3: How to scrape an HTTPS website that has JavaScript by tomhukins (Curate) on Aug 23, 2004 at 17:08 UTC
It was a few months ago, but I recall successfully using HTTP::Recorder to use HTTP::Proxy with JavaScript enabled sites. HTTP::Proxy traps all HTTP requests, regardless of whether they are plain HTML hyperlinks or initiated by JavaScript. Granted, this only works when JavaScript does something simple, but in my experience it usually does. If the JavaScript does anything complex, then you're right: you'll have to either run a JavaScript interpreter within Perl or rewrite the algorithm in Perl.	[reply]
Re^3: How to scrape an HTTPS website that has JavaScript by perrin (Chancellor) on Aug 23, 2004 at 19:17 UTC
I did let the author know. She said that it kind of depends on what your JavaScript does. However, this is more the case if you are trying to use HTTP::Recorder to generate QA scripts that test your JavaScript. I don't think there is anything JavaScript could do that would matter when scraping a site which would not be captured by an HTTP proxy.	[reply]