Unfortunately, with robots.txt being global, this is pointless. However, the engines that power Google, Yahoo, and the other majors are well known, and it would be beneficial IMHO to allow them to do what they do. I'm glad to see that some of the pieces are there, thanks to your efforts, planetscape. :D