Re^10: Super search use DuckDuckGo link broken (excluding special pages and noise)

That's true, but nofollow is - according to Wikipedia - a misnomer.

The main purpose is to influence page rankings, especially for links posted by users.

Bots have no obligation not to follow.

I will later run tests with various search engines, to see if "displaytype" is included in search results for the main domain www.perlmonks.org (for the other domains engines may still show cached results)

If not, we are good.

Cheers Rolf
_{(addicted to the Perl Programming Language :)

see Wikisyntax for the Monastery}

Comment on Re^10: Super search use DuckDuckGo link broken (excluding special pages and noise) Select or Download Code

Replies are listed 'Best First'.
Re^11: Super search use DuckDuckGo link broken (excluding special pages and noise) by Corion (Patriarch) on May 04, 2025 at 17:12 UTC
Oh, thanks, I had the wrong impression of these attributes then! I've added the following lines to `robots.txt`, we'll see if bots honor these: `Disallow: ?displaytype= Disallow: &displaytype= Disallow: ?node=Super+Search Disallow: &node=Super+Search Disallow: ?node_id=3989` [download] The main source of requests were crawlers that don't even identify themselves explicitly as bots, so I really doubt that the problematic bots will care about `robots.txt`.	[reply] [d/l] [select]
Re^12: Super search use DuckDuckGo link broken (excluding special pages and noise) by LanX (Saint) on May 07, 2025 at 21:32 UTC
Thanks for updating robots.txt `# Be kind. Wait between fetches longer than each fetch takes. User-agent: * Disallow: /bare/ Disallow: /mobile/ Disallow: /?displaytype= Disallow: /&displaytype= Disallow: /?node=Super+Search Disallow: /&node=Super+Search Disallow: /?node_id=3989 Crawl-Delay: 20` [download] But as I already said, you are forgetting the very common semicolon `Disallow: /;displaytype= Disallow: /;node_id=3989` [download] Personally I'd simplify and skip all `; & ?` `# Be kind. Wait between fetches longer than each fetch takes. User-agent: Disallow: /bare/ Disallow: /mobile/ Disallow: /displaytype= Disallow: /node=Super+Search Disallow: /node_id=3989 Crawl-Delay: 20` [download] The only problem I see are IDs starting with 3989 etc (e.g. 39895) So probably better: `Disallow: /node=Super+Search$ Disallow: /node_id=3989$ Disallow: /node=Super+Search; Disallow: /node_id=3989; Disallow: /node=Super+Search& Disallow: /*node_id=3989&` [download] Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery}	[reply] [d/l] [select]
Re^12: Super search use DuckDuckGo link broken (excluding special pages and noise) by LanX (Saint) on May 04, 2025 at 17:30 UTC
From my understanding one still needs a wildcard `/` up front,( but I'm not sure) Anyway you forgot `;` as separator in searchstrings. I'll suggest just `/displaytype=` it should cover all cases and only searchstrings, since `=` is meta. I thought I read that Google has a page for testing rules online, but I can't find it right now. * FWIW: I read at Google that adbots are not covered by * and must be listed explicitly. But I suppose those are not our biggest headache. On a side note: adding `rel='nofollow'` to links posted by users should help frustrating spammers. Edit *) See https://support.google.com/webmasters/answer/6062598?hl=en Couldn't test it since it requires a log in. Update Forgot to mention, I tried finding `displaytype` nodes with Google and DDG and the nofollow rules seem to be effective on site:www.perlmonks.org. Other domains seem to show cached results. Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery}	[reply] [d/l] [select]

Edit

Update