in reply to Can dynamic sites be parsed by search engine robots?

dont forget the robots.txt, no spiders indexed my site without one.

-- http://www.whackpack.com => FOO!
  • Comment on Re: Can dynamic sites be parsed by search engine robots?

Replies are listed 'Best First'.
RE: Re: Can dynamic sites be parsed by search engine robots?
by cianoz (Friar) on Oct 08, 2000 at 16:31 UTC
    this is not correct:
    robot.txt allow you to restrict/control how robots index your site, is not necessary to make them work!
[reply]

      Absolutes are so dangerous, usually =)

      Many robots act differently if there is an active robots.txt file in place. Many use a more liberal tree walker and will follow some dynamic content if you ask nicely.

      Put in a /robots.txt file and a */* rule.

      --
      $you = new YOU;
      honk() if $you->love(perl)

[reply]
        ok, often i tend to confuse how things should work with how things actually work... :-)
[reply]

In Section Seekers of Perl Wisdom