in reply to Web page scraping from AJAX page with POST JSON data

I'm not sure how to change the Content-Type header with HTTP::Request::Common (for plain WWW::Mechanize), but it should give you at least a HTTP::Request whose Content-Type you can change later on:

use strict; use HTTP::Request::Common; my $request= POST 'http://book.flypeach.com/WebService/B2cService.asmx +/SearchLowFareSingleMonth', Content_Type => 'application/json', Content => '{"strFromAirport":"KIX","strToAirport":"HKG","dep +artMonth":"20150606","returnMonth":"20150606","iOneWay":"true","iAdul +t":2,"iChild":0,"iInfant":0,"BoardingClass":"","CurrencyCode":"JPY"," +strPromoCode":"","SearchType":"FARE","iOther":0,"otherType":"","stIpA +ddress":"","strCurrentDate":"20150406"}', ; print $request->as_string; __END__ POST http://book.flypeach.com/WebService/B2cService.asmx/SearchLowFare +SingleMont h Content-Length: 295 Content-Type: application/json {"strFromAirport":"KIX","strToAirport":"HKG","departMonth":"20150606", +"returnMonth":"20150606","iOneWay":"true","iAdult":2,"iChild":0,"iInf +ant":0,"BoardingClass":"","CurrencyCode":"JPY","strPromoCode":"","Sea +rchType":"FARE","iOther":0,"otherType":"","stpAddress":"","strCurrent +Date":"20150406"}

If you want to keep using WWW::Mechanize::Firefox, note that its support to add custom headers is somewhat limited, but the same approach could still work. But if you already use WWW::Mechanize::Firefox, why not just keep automating the complete website and programmatically click your way to the data you want?

Replies are listed 'Best First'.
Re^2: Web page scraping from AJAX page with POST JSON data
by ronstudio (Novice) on Apr 06, 2015 at 02:09 UTC

    Hi Corion, thanks for your reply!

    I'm just didn't realize I can further navigate the website with WWW:Mechanize:Firefox. I were just keep thinking of if there is any possible way to extract those price data table directly.

    Let me dig further regarding this, thanks very much!!!

    btw, are you the author of http://corion.net/talks/web-scraping-with-perl/web-scraping-with-perl.en.html ? I have read some example in there. Thanks very much for your help and info from that page

      Hi Corion,

      After a good sleep and your advice, I found that I can use the key "data" to represent those JSON value and successfully sending the http request exactly the same as manual browsing.

      In order to get the price table which I need, somehow I need to browse through the site with some sequences of page (first the form dialogue, then sending the json header). At least I can get what I need following your advice to simulate what the browser is doing manually!

      Thanks~~