Parsing AJAX-based website

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Parsing AJAX-based website by grinder (Bishop) on Feb 09, 2008 at 16:25 UTC
How do I simulate this to get to the information I want to extract actually? With great patience and attention to detail. It's just about impossible in the general case. Have a look at the WWW::Mechanize::FAQ, on the section "why don't you support JavaScript?". In a nutshell, you need to reimplement a JavaScript engine, and that's a non-trivial undertaking. The usual way of going about automating a website that makes heavy use of JavaScript is to insert an HTTP::Proxy-based proxy between your client and the server, to record precisely what is being passed back and forth. At the lowest level, it's just GET, POST and following redirects, and doing the same thing in your program. But depending on the site, this can be very difficult to do. • another intruder with the mooring in the heart of the Perl	[reply]
Re^2: Parsing AJAX-based website by Joost (Canon) on Feb 09, 2008 at 17:16 UTC
The javascript engine isn't even that big of a problem. There are at least two modules on CPAN that interface with the spidermonkey JS interpreter (though last time I checked, neither was complete and using them wasn't exactly trivial). The real issue is that you'd have to implement pretty much the whole DOM too, including most of the non-standardized stuff. Which is really, really tedious work. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]
Re: Parsing AJAX-based website by bradcathey (Prior) on Feb 09, 2008 at 16:37 UTC
Take a look at AHAH, a subset of Ajax that is much easier and does much of the same. Also, no XML—I just don't need the power of XML for basic Web forms, so I don't have to deal with the overhead of parsing XML (although there are some CPAN modules to help). I had my first AHAH/Perl example built in about an hour (created a select tag dynamically based on user choices). —Brad "The important work of moving the world forward does not wait to be done by perfect men." George Eliot	[reply]
Re^2: Parsing AJAX-based website by samtregar (Abbot) on Feb 10, 2008 at 20:03 UTC
How is AHAH supposed to help someone trying to scrape an existing AJAX site? -sam	[reply]
Re: Parsing AJAX-based website by samtregar (Abbot) on Feb 10, 2008 at 20:00 UTC
My advice is to ignore the AJAX layer and target the lower-level HTTP requests going from the browser to the server. You can capture these using a proxy like HTTP::Recorder or something like the FireBug plugin. If you're lucky you'll find that underneath the glitzy AJAX there's a relatively simple protocol - the browser POSTs some data (user/pass) and gets back some JSON or XML indicating the result (login succeeded or failed). You can then use WWW::Mechanize to imitate that protocol and extract the info you need. If the site authors did a good job this can actually be easier than scraping an HTML page. Good luck! -sam	[reply]
Re^2: Parsing AJAX-based website by Anonymous Monk on Feb 25, 2013 at 12:48 UTC
I am trying to parse a website and the site loads a basic page that I can get with mechanize-get($url). But the Tags and Page data that I want to get is not found content. I think its an ajax call. I do not know how to get my script to get the tags. When I look at google chrome(element inspect), ie(F12 key), mozilla firebug all of them show the tags that I am looking for. But how do I get to the tags. Help will be appreciated.	[reply]
Re^3: Parsing AJAX-based website by marto (Cardinal) on Feb 25, 2013 at 12:52 UTC
WWW::Mechanize states that it does not support JavaScript. Use something that does. Using WWW::Selenium To Test Or Automate An Ajax Website, WWW::Mechanize::Firefox, Super Search.	[reply]
Re^3: Parsing AJAX-based website by tmharish (Friar) on Feb 25, 2013 at 12:52 UTC
Look at the Net tab in firebug or 'Network' tab on chrome and find all the requests going out. Identify the request you are interested in - find out how that part is constructed ( based on the AJAX code that is present ) and make that call. As a note some of the modules mentioned above do this for you.	[reply]