I made a small change so that search engine spiders would no longer be indexing (and caching) public chatter (at least in the most obvious way). We had bigger plans for more improvements directed at providing more usefully tailored content to web spiders, particularly search engine spiders, but it doesn't look like that is going happen any time soon.

So, if you are disappointed to find that your personal web spider is only seeing

and the web crawler heard nothing...
in the place where it used to show public chatter, than this is the thread for you.

Even non-search-engine web spiders should not be fetching public chatter via fetching regular pages (we have pages, too many in fact, specifically for the fetching of public chatter). And, for example, if you have a spider grabbing pages for you to read off-line, then the by-then stale public chatter is of questionable value anyway.

Spiders that provide valid cookies (or use other means) to fetch pages as a logged-in user will not be affected by this change. So, if you are affected and want your spider to fetch pages with public chatter in the nodelet again, then you can arrange for your spider to log in.

Another solution is to try to disguise your spider. If you are tempted to do that, then please reply in this thread instead so we can come up with a simpler solution for spiders to declare that they want to see public chatter. Although I can't currently come up with any good reason for anonymous spiders to have much interest in public chatter, I expect someone will provide a reason here soon enough.

- tye        

In reply to Public chatter is no longer for web crawlers by tye

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.