in reply to Re^11: Super search use DuckDuckGo link broken (excluding special pages and noise)
in thread Super search use DuckDuckGo link broken
# Be kind. Wait between fetches longer than each fetch takes. User-agent: * Disallow: /bare/ Disallow: /mobile/ Disallow: /*?displaytype= Disallow: /*&displaytype= Disallow: /*?node=Super+Search Disallow: /*&node=Super+Search Disallow: /*?node_id=3989 Crawl-Delay: 20
But as I already said, you are forgetting the very common semicolon
Disallow: /*;displaytype= Disallow: /*;node_id=3989
Personally I'd simplify and skip all ; & ?
# Be kind. Wait between fetches longer than each fetch takes. User-agent: * Disallow: /bare/ Disallow: /mobile/ Disallow: /*displaytype= Disallow: /*node=Super+Search Disallow: /*node_id=3989 Crawl-Delay: 20
The only problem I see are IDs starting with 3989 etc (e.g. 39895)
So probably better:
Disallow: /*node=Super+Search$ Disallow: /*node_id=3989$ Disallow: /*node=Super+Search; Disallow: /*node_id=3989; Disallow: /*node=Super+Search& Disallow: /*node_id=3989&
Cheers Rolf
(addicted to the Perl Programming Language :)
see Wikisyntax for the Monastery
|
|---|