in reply to Spidering CGI

If you're using Apache mod_rewrite is very useful for handling this sort of thing.

You can easily change any URL under the hood without the users knowledge so they would see:

http://www.foo.com/foo/bar/index.html

and your code would see

http://www.foo.com/foo/bar/index.cgi

without the user knowing anything about index.cgi.