in reply to Price finding...

You are aware that the terms of use of Amazon include the following:
This license does not include (...) any use of data mining, robots, or similar data gathering and extraction tools.
And eBay does not allow you to
bypass our robot exclusion headers or other measures we may use to prevent or restrict access to the sites.
But eBay's robots.txt is as follows:
### BEGIN FILE ### # # allow-all # # # The use of robots or other automated means to access the eBay site # without the express permission of eBay is strictly prohibited. # Notwithstanding the foregoing, eBay may permit automated access to # access certain eBay pages but soley for the limited purpose of # including content in publicly available search engines. Any other # use of robots or failure to obey the robots exclusion standards set # forth at <http://www.robotstxt.org/ wc/ exclusion.html> is strictly # prohibited. # User-agent: eBay-crawler Disallow: User-agent: * Disallow: /disney/ ### END FILE ###
As can readily be seen, this robots.txt is really broken: The text part of it does not allow web-scraping and only very limited use of crawling when permission is granted, but its rules only disallow crawler-access to the /disney/ pages (copyright and licensing issues no doubt) for all web-crawlers and the eBay-crawler can go anywhere. So there is a incompatibility between the text and the rules. I'd say that in this case, the rules win over the text since the robot is not required to read (and act upon) the text. Still, it is very sloppy from eBay to publish such a file. On the other hand, there is a finality build into the text: even if your "robot" is allowed access he may only do so "for the limited purpose of including content in publicly available search engines" and all other use or automated access is forbidden. And there is something to say for the argument that this is directed at a human and not at a computer. So, there is a risk that your scraping of eBay is wrong, especially if you specifically target their site and not stumble upon it as a "dumb" crawler is likely to do.

Update: You may find it silly that these sites forbid you to scrape their web-pages but on the other hand have a published API to get you the same (and even more) info (than) you can scrape from your screen. They may have very good reasons to force you into using the published API as it almost guarantees you get "good data", they can control who is allowed to use this data (you must apply for a user ID) and is generally "cheaper" on their resources. So I advise you to get a licence to be allowed to use the API and then use that.

CountZero

A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James