Re^3: [OT] Ethical and Legal Screen Scraping

You seem to be confused about ethics. Behaving ethically is not defined by anyone else's ability to prove that you did or did not behave ethically.

Looks like you didn't understand me. You're confusing the "ethic" aspect of my answer with the "legal" aspect. I mean "enforcing legally the respect of robots.txt is abusive and unethical.". I mean too "I can't imagine that you'd be legally in trouble for web-spidering, for personal use, without taking care of robots.txt, except in China or Iran.". I never meant "Go suck all websites you want to and don't bother", ever.

I think too the generalisation "if you use a web-spidering program without taking notice of robots.txt, you're ethically wrong", is questionable. Ethics aren't that simple. Perhaps he's a political refugee trying to extract important information from some hostile hidden website. Is he "right" or "wrong"?

Comment on Re^3: [OT] Ethical and Legal Screen Scraping

Replies are listed 'Best First'.
Re^4: [OT] Ethical and Legal Screen Scraping by tilly (Archbishop) on Jul 26, 2005 at 22:44 UTC
I had misunderstood that you were saying that the law itself is unethical. I think that distinction was probably lost on many others as well. However, as I think the subsequent discussion clarified, my views are not as simple as saying that thou shalt always do what robots.txt says. (Paying attention to it does not mean that you necessarily agree or do what it says...)	[reply]
Re^5: [OT] Ethical and Legal Screen Scraping by wazoox (Prior) on Jul 27, 2005 at 05:32 UTC
Well, I probably wasn't clear enough myself. And fortunately it's not yet exactly a law but a jurisprudence AFAIK, which is somewhat better (if a law comes someday to draw the line...)	[reply]