You are aware that the terms of use of Amazon include the following:

This license does not include (...) any use of data mining, robots, or similar data gathering and extraction tools.

And eBay does not allow you to

bypass our robot exclusion headers or other measures we may use to prevent or restrict access to the sites.

But eBay's robots.txt is as follows:

### BEGIN FILE ###
#
# allow-all
#
#
# The use of robots or other automated means to access the eBay site
# without the express permission of eBay is strictly prohibited.
# Notwithstanding the foregoing, eBay may permit automated access to
# access certain eBay pages but soley for the limited purpose of
# including content in publicly available search engines. Any other
# use of robots or failure to obey the robots exclusion standards set
# forth at <http://www.robotstxt.org/ wc/ exclusion.html> is strictly
# prohibited.
#
User-agent: eBay-crawler
Disallow:


User-agent: *
Disallow: /disney/

### END FILE ###
[download]

As can readily be seen, this robots.txt is really broken: The text part of it does not allow web-scraping and only very limited use of crawling when permission is granted, but its rules only disallow crawler-access to the /disney/ pages (copyright and licensing issues no doubt) for all web-crawlers and the eBay-crawler can go anywhere. So there is a incompatibility between the text and the rules. I'd say that in this case, the rules win over the text since the robot is not required to read (and act upon) the text. Still, it is very sloppy from eBay to publish such a file. On the other hand, there is a finality build into the text: even if your "robot" is allowed access he may only do so "for the limited purpose of including content in publicly available search engines" and all other use or automated access is forbidden. And there is something to say for the argument that this is directed at a human and not at a computer. So, there is a risk that your scraping of eBay is wrong, especially if you specifically target their site and not stumble upon it as a "dumb" crawler is likely to do.

Update: You may find it silly that these sites forbid you to scrape their web-pages but on the other hand have a published API to get you the same (and even more) info (than) you can scrape from your screen. They may have very good reasons to force you into using the published API as it almost guarantees you get "good data", they can control who is allowed to use this data (you must apply for a user ID) and is generally "cheaper" on their resources. So I advise you to get a licence to be allowed to use the API and then use that.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.