The robots or crawlers are free to fully disregard the robots.txt directives. Certainly that is not nice, but the world is full of less than nice people (and robots and crawlers and ...)

And conversely, there are less than nice web sites blocking robots for no reason.