in reply to Web page scraping from AJAX page with POST JSON data
I'm not sure how to change the Content-Type header with HTTP::Request::Common (for plain WWW::Mechanize), but it should give you at least a HTTP::Request whose Content-Type you can change later on:
use strict; use HTTP::Request::Common; my $request= POST 'http://book.flypeach.com/WebService/B2cService.asmx +/SearchLowFareSingleMonth', Content_Type => 'application/json', Content => '{"strFromAirport":"KIX","strToAirport":"HKG","dep +artMonth":"20150606","returnMonth":"20150606","iOneWay":"true","iAdul +t":2,"iChild":0,"iInfant":0,"BoardingClass":"","CurrencyCode":"JPY"," +strPromoCode":"","SearchType":"FARE","iOther":0,"otherType":"","stIpA +ddress":"","strCurrentDate":"20150406"}', ; print $request->as_string; __END__ POST http://book.flypeach.com/WebService/B2cService.asmx/SearchLowFare +SingleMont h Content-Length: 295 Content-Type: application/json {"strFromAirport":"KIX","strToAirport":"HKG","departMonth":"20150606", +"returnMonth":"20150606","iOneWay":"true","iAdult":2,"iChild":0,"iInf +ant":0,"BoardingClass":"","CurrencyCode":"JPY","strPromoCode":"","Sea +rchType":"FARE","iOther":0,"otherType":"","stpAddress":"","strCurrent +Date":"20150406"}
If you want to keep using WWW::Mechanize::Firefox, note that its support to add custom headers is somewhat limited, but the same approach could still work. But if you already use WWW::Mechanize::Firefox, why not just keep automating the complete website and programmatically click your way to the data you want?
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: Web page scraping from AJAX page with POST JSON data
by ronstudio (Novice) on Apr 06, 2015 at 02:09 UTC | |
by ronstudio (Novice) on Apr 06, 2015 at 04:23 UTC |