G'day doubledecker,

"Also, I noticed 'robots' in the source code which I never heard of."

Providing a single word with no context is very vague. Was it in markup like this from W3C? If so, then http://www.robotstxt.org/robotstxt.html probably has the information you want (or provides links to it).