Absolutes are so dangerous, usually =)

Many robots act differently if there is an active robots.txt file in place. Many use a more liberal tree walker and will follow some dynamic content if you ask nicely.

Put in a /robots.txt file and a */* rule.

--
$you = new YOU;
honk() if $you->love(perl)