For future reference, if you code up an example robot, please show how to fetch robots.txt and use http://WWW::RobotRules.

It is unlikely that the person asking will know that this is important, and therefore it is our responsibility to pass along best practices.