in reply to Or spin the bot back?
in thread How to implement a fourth protocol
Spiders don't revisit pages they have already visited. That's necessary because web sites are naturally full of loops. For example, the home page links to the contact page, and the contact page links to the home page. Since it doesn't make the same request twice, a client-side cache will never get hit. You can make the client spin, but it will involve a request to your server.
|
|---|