in reply to Re^11: Super search use DuckDuckGo link broken (excluding special pages and noise)
in thread Super search use DuckDuckGo link broken

Thanks for updating robots.txt
# Be kind. Wait between fetches longer than each fetch takes. User-agent: * Disallow: /bare/ Disallow: /mobile/ Disallow: /*?displaytype= Disallow: /*&displaytype= Disallow: /*?node=Super+Search Disallow: /*&node=Super+Search Disallow: /*?node_id=3989 Crawl-Delay: 20

But as I already said, you are forgetting the very common semicolon

Disallow: /*;displaytype= Disallow: /*;node_id=3989

Personally I'd simplify and skip all ; & ?

# Be kind. Wait between fetches longer than each fetch takes. User-agent: * Disallow: /bare/ Disallow: /mobile/ Disallow: /*displaytype= Disallow: /*node=Super+Search Disallow: /*node_id=3989 Crawl-Delay: 20

The only problem I see are IDs starting with 3989 etc (e.g. 39895)

So probably better:

Disallow: /*node=Super+Search$ Disallow: /*node_id=3989$ Disallow: /*node=Super+Search; Disallow: /*node_id=3989; Disallow: /*node=Super+Search& Disallow: /*node_id=3989&

Cheers Rolf
(addicted to the Perl Programming Language :)
see Wikisyntax for the Monastery