in reply to Re: CB history and Google's cache (fixed)
in thread CB history and Google's cache
PM's /robots.txt says:
# sorry, but misbehaved robots have ruined it for all of you.so google shouldn't be indexing PM directly at all. But I believe that google still does index PM to some extent, perhaps when prompted to by people giving specific URLs to less-obvious google tools (which could be construed as not being "robot" traffic) -- though I'm just guessing wildly here.
User-agent: *
Disallow: /
Corion is working on building perlmonks.org/robot/... and then we'll tell robots (via robots.txt) to only scan that part of the site and we'll also detect common robots (via user agent string) and redirect them there as well.
- tye
|
|---|