in reply to [OT] Ethical and Legal Screen Scraping
If a commercial company sells a tool that allows non-programmer end users to write little screen scraping robots, is it unethical or illegal for such a product to not provide a mechanism to allow their end users to respect robots.txt?First of all, I don't think the answer to the question depends on whether the tool is sold or not, or whether it's produced by a company that may, or may not, be commercial.
As for the legal status, like many other things on the internet, that will remain unclear for a long time. Jurisdiction concerning internet matters grows very slowly - the fact the internet doesn't care about our legal borders is a not unimportant factor in the slow growth of jurisdiction.
As for the ethical thing. If it doesn't allow a mechanism to respect robots.txt, it's certainly a bad product. And I guess selling bad products is unethical. It's certainly unethical for the end user to use such a tool though. (Well, IMO). It's the user of robot created that way that carries the responsibility - just like you carry the responsibility your car has headlights and brakes, the judge won't accept a defense of "but it was sold to me this way".
|
|---|