I made a small change so that search engine spiders would no longer be indexing (and caching) public chatter (at least in the most obvious way). We had bigger plans for more improvements directed at providing more usefully tailored content to web spiders, particularly search engine spiders, but it doesn't look like that is going happen any time soon.

So, if you are disappointed to find that your personal web spider is only seeing

and the web crawler heard nothing...

in the place where it used to show public chatter, than this is the thread for you.

Even non-search-engine web spiders should not be fetching public chatter via fetching regular pages (we have pages, too many in fact, specifically for the fetching of public chatter). And, for example, if you have a spider grabbing pages for you to read off-line, then the by-then stale public chatter is of questionable value anyway.

Spiders that provide valid cookies (or use other means) to fetch pages as a logged-in user will not be affected by this change. So, if you are affected and want your spider to fetch pages with public chatter in the nodelet again, then you can arrange for your spider to log in.

Another solution is to try to disguise your spider. If you are tempted to do that, then please reply in this thread instead so we can come up with a simpler solution for spiders to declare that they want to see public chatter. Although I can't currently come up with any good reason for anonymous spiders to have much interest in public chatter, I expect someone will provide a reason here soon enough.