in reply to robots.txt, google, et al

This is not strictly true - just recently, tye made the appropriate changes to allow Google back into the site. All other search engines are still blocked though, because at least one of them was badly misbehaving.

Update: Indeed, www.perlmonks.org is the domain that's open again, all other domains are blocked for spiders.

Replies are listed 'Best First'.
Re^2: robots.txt, google, et al
by Limbic~Region (Chancellor) on Mar 07, 2007 at 17:28 UTC
    Corion,
    All other search engines are still blocked though,...

    I think you have your facts mixed up. What is being blocked is which domain names can be spidered. The robots.txt file that does allow spidering allows all user agents. See my note in this thread for details.

    Cheers - L~R