Re^7: Super search use DuckDuckGo link broken

Yes, see the corresponding robots.txts

# Please only spider https://www.perlmonks.org not https://perlmonks.c
+om
User-agent: *
Disallow: /
[download]

https://www.perlmonks.org/robots.txt

# Be kind. Wait between fetches longer than each fetch takes.
User-agent: *
Disallow: /bare/
Disallow: /mobile/
Crawl-Delay: 20
[download]

Cheers Rolf
_{(addicted to the Perl Programming Language :)

see Wikisyntax for the Monastery}

Comment on Re^7: Super search use DuckDuckGo link broken Select or Download Code

Replies are listed 'Best First'.
Re^8: Super search use DuckDuckGo link broken (excluding special pages and noise) by LanX (Saint) on May 04, 2025 at 14:10 UTC
According to the robots.txt specifications I found at Google, it's possible to exclude "orthogonal" pages like `&displaytype=print` or `;displaytype=edithistory` with wildcards. Any reason not to add `Disallow: /*displaytype=` to the list? (Untested) Bing also suggests adding noindex-meta tags to the pages. On a tangent Ideally robots would be presented with a page without nodelets, but I'm not aware of an efficient solution, except checking the user-agent before building the page. Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery}	[reply] [d/l] [select]
Re^9: Super search use DuckDuckGo link broken (excluding special pages and noise) by Corion (Patriarch) on May 04, 2025 at 14:38 UTC
These links are already tagged with `rel="nofollow"`, so Google shouldn't be crawling these, nor should any other bots. Except a lot of them do, so I'm not sure if spending any effort on divining bot behaviours is time well spent.	[reply] [d/l]
Re^10: Super search use DuckDuckGo link broken (excluding special pages and noise) by LanX (Saint) on May 04, 2025 at 14:59 UTC
That's true, but `nofollow` is - according to Wikipedia - a misnomer. The main purpose is to influence page rankings, especially for links posted by users. Bots have no obligation not to follow. I will later run tests with various search engines, to see if "displaytype" is included in search results for the main domain `www.perlmonks.org` (for the other domains engines may still show cached results) If not, we are good. Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery}	[reply] [d/l] [select]
Re^11: Super search use DuckDuckGo link broken (excluding special pages and noise) by Corion (Patriarch) on May 04, 2025 at 17:12 UTC
Re^12: Super search use DuckDuckGo link broken (excluding special pages and noise) by LanX (Saint) on May 07, 2025 at 21:32 UTC
Re^12: Super search use DuckDuckGo link broken (excluding special pages and noise) by LanX (Saint) on May 04, 2025 at 17:30 UTC
Re^10: Super search use DuckDuckGo link broken (excluding special pages and noise) by jdporter (Paladin) on May 05, 2025 at 12:36 UTC
Please see subthread at Re^8: Unable to connect, if you haven't already. (And I acknowledge that you might find no new info there.)	[reply]