in reply to Can dynamic sites be parsed by search engine robots?

footpad,
if you registered your site, a bot will try to open it, as he would do with any available domain, e.g. "www.mypersonalweb.com/personal/info/about/me/"
would do it, but notice, that if your provider uses apache, it might happen that the client browser (or in this case the bot) get's an "acces denied", if the trailing slash is omitted when the url is not pointing to the domain root.
So first you should check with a browser, if you get the document specified by the URL you registered at the search engine.
Normally the search bots will look for meta tags for robots, where you might specify if a search bot shall follow up the links in this page, with "follow" or "nofollow".
It is important to notice that this concerns not the directory structure of your web, cause the bots are normal HTTP-clients.
So you might also check, if in your sites root directory is a "robots.txt" file as well :-)
To check if a specified client is requesting your webserver to get this document, you might look at this one of merlyn's columns for webtechniques where he describes how to generate acces logs. While reading through this code you will easily be able to figure out, which four lines of code to include into your cgi-script, that is responding to the HTTP_requests for the document specified by the URL you registered.
Sorry for "talking around the corner", but my English is not as good as it should be.
Have a nice day
All decision is left to your taste
  • Comment on RE: Can dynamic sites be parsed by search engine robots?