How do I Extract Data From Amazon.com's Website?

Perl15 has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: How do I Extract Data From Amazon.com's Website? by FunkyMonk (Bishop) on Aug 15, 2007 at 15:28 UTC
Net::Amazon seems to be up-to-date and comprehensive package for working with Amazon. Do any of the examples in the documentation help?	[reply] [d/l] [select]
Re: How do I Extract Data From Amazon.com's Website? by barbie (Deacon) on Aug 15, 2007 at 16:19 UTC
I realise it's blowing my own trumpet, but have a look at WWW-Scraper-ISBN-Amazon_Driver too :) -- Barbie \| Birmingham Perl Mongers \| CPAN Testers Wiki \| CPAN Testers Statistics	[reply]
Re: How do I Extract Data From Amazon.com's Website? by renodino (Curate) on Aug 15, 2007 at 17:34 UTC
DBD::Amazon might be a solution if you're comfortable with SQL and your resultset isn't as massive as such a request can generate. The query engine attempts to optimize by extracting predicates that can be filtered by Amazon, and IIRC Publisher is one of those. You'll need to get an ECS account, however. BTW: You might want to be a bit more specific in your post's title. Perl Contrarian & SQL fanboy	[reply]
Re: How do I Extract Data From Amazon.com's Website? by dwm042 (Priest) on Aug 15, 2007 at 17:52 UTC
Scraping information off web sites.. books have been written on the topic. Other authors have mentioned specifics. You can Google screen scraping for more information, and this O' Reilly book I've found helpful in many ways. Most of the book's examples are Perl code. Spidering Hacks	[reply]
Re: How do I Extract Data From Amazon.com's Website? by matsi (Novice) on Aug 15, 2007 at 18:43 UTC
Respected Monks suggested several modules aimed at handling Amazon site. But if you want implement similar stuff yourself or probably you need to solve similar problem for another website, then you should look around LWP modules.	[reply]
Re^2: How do I Extract Data From Amazon.com's Website? by Joost (Canon) on Aug 15, 2007 at 18:50 UTC
If you're dealing with a web interface (i.e. HTML with forms, links, etc instead of "pure" HTTP) you're probably better off using WWW::Mechanize - it's an LWP subclass with lots of specialized methods to search and "click" through web pages/forms. There are even a few "clones" of WWW::Mechanize that use popular browsers at the back-end so you can deal with javascript and other client-side objects not normally supported by WWW::Mechanize. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]
Re^3: How do I Extract Data From Amazon.com's Website? by erroneousBollock (Curate) on Aug 16, 2007 at 06:58 UTC
Recently, some folks figured out a way to use Mozilla in a mechanize driver without the need for a (visible) X-server. Sounds promising. -David	[reply]
Re: How do I Extract Data From Amazon.com's Website? by Philem (Acolyte) on Aug 15, 2007 at 19:17 UTC
Additionally, you may want to have a look at http://samie.sourceforge.net. I've used it in the past and it's very good, once you get used to it.	[reply]
Re: How do I Extract Data From Amazon.com's Website? by eriam (Beadle) on Aug 16, 2007 at 06:48 UTC
Hello, A lot of good tools has been mentionned already but I'd like to mention also Template-Extract ! And of course POE is the way to go when it comes to networking latency exposed applications. Thank you Eriam	[reply]