How do I perform web automation with sites that use Javascript?

whakka has asked for the wisdom of the Perl Monks concerning the following question: ⭐ (http and ftp clients)

Replies are listed 'Best First'.
Re: How do I perform web automation with sites that use Javascript?⭐ by jettero (Monsignor) on May 02, 2009 at 19:12 UTC
JavaScript::SpiderMonkey seems to be in use by scripts built as recently as Net::Plurk::Dumper. Something to add to the list in any case.	[reply]
Re: How do I perform web automation with sites that use Javascript?⭐ by jdporter (Paladin) on May 04, 2009 at 18:10 UTC
Here are some modules which give you a way around the issue: WWW::Scripter WWW::Selenium Win32::IE::Mechanize Mozilla::Mechanize Other things to try: Disable Javascript in your browser and see if the site still functions as you want. If so, then you don't actually have a problem :) Figure out what the scripts are doing on the wire, and re-implement those transactions in your own program. The Firefox add-on Live HTTP Headers is well suited for this. This info provided by the OP.	[reply]
Re: How do I perform web automation with sites that use Javascript?⭐ by planetscape (Chancellor) on May 04, 2009 at 05:31 UTC
Don't forget Limbic~Region's excellent Tutorial, Using WWW::Selenium To Test Or Automate An Ajax Website.	[reply]
Re: How do I perform web automation with sites that use Javascript? by ninuzzo (Novice) on Mar 12, 2011 at 23:16 UTC
A recent addition of mine is WWW::HtmlUnit::Spidey. This module uses the Java library HtmlUnit which is a headless browser with pretty good JavaScript support. Do not worry, you won't have to write any Java code :D It is good for massive web scraping where screen scraping does not scale and may be unstable. There is a tutorial here that scrapes some data obtained from a form not working without JavaScript support. Btw I am just a Perl beginner. Any Perl guru interested in co-developing Spidey?	[reply]
Re: Answer: How do I perform web automation with sites that use Javascript? by Anonymous Monk on Mar 13, 2011 at 03:59 UTC
If you wish to recruit, try perl news section, much more visible	[reply]
Re: How do I perform web automation with sites that use Javascript? by jdporter (Paladin) on Jun 08, 2009 at 18:35 UTC
Re: How do I perform web automation with sites that use Javascript? Originally posted as a Categorized Answer.	[reply]