We could at least render some kind of very lightly marked up html for the search engines; but it's relatively shocking that the PM robots.txt tells all search engines to go away.

It may have made sense in 1999 (or whenever that went into effect) but not in 2007.

Replies are listed 'Best First'.
Re: robots.txt, google, et al
by Corion (Patriarch) on Mar 07, 2007 at 16:59 UTC

    This is not strictly true - just recently, tye made the appropriate changes to allow Google back into the site. All other search engines are still blocked though, because at least one of them was badly misbehaving.

    Update: Indeed, www.perlmonks.org is the domain that's open again, all other domains are blocked for spiders.

[reply]
      Corion,
      All other search engines are still blocked though,...

      I think you have your facts mixed up. What is being blocked is which domain names can be spidered. The robots.txt file that does allow spidering allows all user agents. See my note in this thread for details.

      Cheers - L~R

[reply]
Re: robots.txt, google, et al
by Limbic~Region (Chancellor) on Mar 07, 2007 at 17:25 UTC
[reply]

Back to Perl Monks Discussion