in reply to Spidering CGI
This is a similar problem to that faced by perlmonks.org which does not have pages a such - it is all database driven. The problem can be solved by putting up real html pages to be spidered like PM does see: http://perlmonks.thepen.com/
Search engines such as google are spidering CGI based sites but have issues, see this article
If you are requiring .htaccess authentication to get to your pages then search spiders will not be able to access them but this hardly matters as nor would anyone following a link.
Presumably you have set your robots files....
cheers
tachyon
s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print
|
|---|