The question is ... does this really matter? I mean if your pages are indexed they are more likely to be found, therefore you get more hits, more ad views and in the end more money. So I would not care that much if a search engine flooded my server with requests once a month.

So IMHO the only violation that might matter is not obeying robots.txt. Actually could someone give me some example of a reasonable robots.txt usage?

Jenda
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
   -- Rick Osborne

Edit by castaway: Closed small tag in signature