Re^2: Searching the monastery with duckduckgo leads to ugly results

Thanks.

For the record:

I took a look at the robots.txt and they don't seem to do much except blocking any combination of www.com | www.net | .org to only allow www.perlmonks.org

# Please only spider http://www.perlmonks.org not http://perlmonks.org
User-agent: *
Disallow: /
[download]

I couldn't find any rules blocking /bare or /mobile

There are also at least two other pair domains m/qs\d+.pair.com/ showing up, which seem to be (have been?) used for development and have no robots.txt at all to block them.

edit

FWIW: there is also the separate issue of blocking ?displaytype= like xml or print , but I'm not sure if there is an accepted standard to block on /?searchstring patterns.

Update

https://en.wikipedia.org/wiki/Robots_exclusion_standard

We could also use meta-tags to disallow print versions

Cheers Rolf
_{(addicted to the Perl Programming Language :)

Wikisyntax for the Monastery}

Comment on Re^2: Searching the monastery with duckduckgo leads to ugly results Select or Download Code

Replies are listed 'Best First'.
Re^3: Searching the monastery with duckduckgo leads to ugly results (WWW::RobotRules) by LanX (Saint) on Nov 01, 2021 at 12:25 UTC
> but I'm not sure if there is an accepted standard to block Fun: ddg seems to use a Perl module WWW::RobotRules for blocking rules. :) Details: https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery}	[reply]


Problems? Is your data what you think it is?
	PerlMonks