I think the problem is that its not unlikely that a web crawler ends up hammering the system following way way too many links generating zillions of unnecessary page fetches. For instance, the spider lands on the front page. That has links to the high activity sections along with a large number of root level nodes. So then it follows each section and each front paged node. Now each one of those nodes has links to itself. So it wont just index the root node and the thread below, but rather the whole thing and then each one singly. Quite possibly it will do this twice for the front paged nodes. Then it will also index each users home node, which of course leads to lists of nodes written by that author which I imagine will eventually result in google single handedly fetching pretty close to each and every node on the site. This is a load that we just dont need.
Of course the CB and various other bits that we dont really want indexed is also a reason. But i should think the core reason is that our site isnt particularly amenable to automated crawlers. The whole point of the blakems static mirror is that it is static and updated rarely and at a low load threshold. Once its mirrored Google can search and index it as it likes, there wont be unnecessary load on our DB servers so we dont really care then.
In reply to Re^4: Google indexes Perlmonks
by demerphq
in thread Google indexes Perlmonks
by bart
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |