in reply to Infinite loop prevention for spider
I did this to see what kind of similar-duplicate rejection algorithms the big indexing spiders use. Most of them recognized rather quickly that the pages were duplicate pages, but NorthernLights had indexed about 20 levels deep of the same pages before I turned the link off. Bleh!
-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.
|
|---|