Re^2: Web page scraping from AJAX page with POST JSON data

Hi Corion, thanks for your reply!

I'm just didn't realize I can further navigate the website with WWW:Mechanize:Firefox. I were just keep thinking of if there is any possible way to extract those price data table directly.

Let me dig further regarding this, thanks very much!!!

btw, are you the author of http://corion.net/talks/web-scraping-with-perl/web-scraping-with-perl.en.html ? I have read some example in there. Thanks very much for your help and info from that page

Comment on Re^2: Web page scraping from AJAX page with POST JSON data

Replies are listed 'Best First'.
Re^3: Web page scraping from AJAX page with POST JSON data by ronstudio (Novice) on Apr 06, 2015 at 04:23 UTC
Hi Corion, After a good sleep and your advice, I found that I can use the key "data" to represent those JSON value and successfully sending the http request exactly the same as manual browsing. In order to get the price table which I need, somehow I need to browse through the site with some sequences of page (first the form dialogue, then sending the json header). At least I can get what I need following your advice to simulate what the browser is doing manually! Thanks~~	[reply]

Replies are listed 'Best First'.

Re^3: Web page scraping from AJAX page with POST JSON data
by ronstudio (Novice) on Apr 06, 2015 at 04:23 UTC

Hi Corion,

After a good sleep and your advice, I found that I can use the key "data" to represent those JSON value and successfully sending the http request exactly the same as manual browsing.

In order to get the price table which I need, somehow I need to browse through the site with some sequences of page (first the form dialogue, then sending the json header). At least I can get what I need following your advice to simulate what the browser is doing manually!

Thanks~~

[reply]