IIRC, something was added to specify that perlmonks.org or www.perlmonks.org is the official domain, so no surprise you get better result from that one.

Yes, see the corresponding robots.txts

Like

• https://perlmonks.com/robots.txt

  # Please only spider https://www.perlmonks.org not https://perlmonks.c
  +om
  User-agent: *
  Disallow: /
  [download]

• https://www.perlmonks.org/robots.txt

  # Be kind. Wait between fetches longer than each fetch takes.
  User-agent: *
  Disallow: /bare/
  Disallow: /mobile/
  Crawl-Delay: 20
  [download]

Cheers Rolf
_{(addicted to the Perl Programming Language :)
see Wikisyntax for the Monastery}

According to the robots.txt specifications I found at Google, it's possible to exclude "orthogonal" pages like &displaytype=print or ;displaytype=edithistory with wildcards.

Any reason not to add

• Disallow: /*displaytype=

to the list? (Untested)

Bing also suggests adding noindex-meta tags to the pages.

On a tangent

Ideally robots would be presented with a page without nodelets, but I'm not aware of an efficient solution, except checking the user-agent before building the page.

Cheers Rolf
_{(addicted to the Perl Programming Language :)
see Wikisyntax for the Monastery}