in reply to Searching the monastery with duckduckgo leads to ugly results

G'day Rolf,

DDG is also my default search engine. I followed your "ddg:" link. Of the first six results, 1-4 & 6 had www.perlmonks.org/bare/...; the 5th was an SO link; there were no further PM links on the first page of results; I didn't look any further.

Under the search query text field (correctly showing "schwartzian transform perlmonks") I see: "All Regions" and "Any Time".

I did notice a minor redirection. Your link has https://duckduckgo.com/html/?q=schwartzian transform perlmonks; my address bar was showing https://html.duckduckgo.com/html/?q=schwartzian transform perlmonks.

I'm using Firefox 93.0 (64-bit) on MS Windows 10 (with latest updates as of three days ago). I don't have any special DDG settings configured.

Being unable to reproduce your results, I can't really comment further; however, sometimes a null result can be useful (e.g. how do your browser, platform, versions and settings differ from mine).

— Ken

Replies are listed 'Best First'.
Re^2: Searching the monastery with duckduckgo leads to ugly results
by LanX (Saint) on Nov 01, 2021 at 11:59 UTC
    Thanks.

    For the record:

    I took a look at the robots.txt and they don't seem to do much except blocking any combination of www.com | www.net | .org to only allow www.perlmonks.org

    # Please only spider http://www.perlmonks.org not http://perlmonks.org
User-agent: *
Disallow: /

    [download]

    I couldn't find any rules blocking /bare or /mobile

    There are also at least two other pair domains m/qs\d+.pair.com/ showing up, which seem to be (have been?) used for development and have no robots.txt at all to block them.

    edit

    FWIW: there is also the separate issue of blocking ?displaytype= like xml or print , but I'm not sure if there is an accepted standard to block on /?searchstring patterns.

    Update

    https://en.wikipedia.org/wiki/Robots_exclusion_standard

    We could also use meta-tags to disallow print versions

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

[reply]
[d/l]
[select]
[reply]

In Section Perl Monks Discussion