in reply to robots.txt, google, et al
This is not strictly true - just recently, tye made the appropriate changes to allow Google back into the site. All other search engines are still blocked though, because at least one of them was badly misbehaving.
Update: Indeed, www.perlmonks.org is the domain that's open again, all other domains are blocked for spiders.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: robots.txt, google, et al
by Limbic~Region (Chancellor) on Mar 07, 2007 at 17:28 UTC |