Re: Searching the monastery with duckduckgo leads to ugly results

I don't really understand the findings.

Is there anything actionable here? Should we block certain? known? bots from visiting certain pages?

Comment on Re: Searching the monastery with duckduckgo leads to ugly results

Replies are listed 'Best First'.
Re^2: Searching the monastery with duckduckgo leads to ugly results by LanX (Saint) on Nov 01, 2021 at 14:13 UTC
Hi I'm still hoping/waiting for input from someone experienced. From what I've read so far: On `www.perlmonks.org` (the main target) `disallow /bare/` `disallow /mobile/` On `qs\d+.pair.com` domains `disallow /` Or `disallow /~perl2/` (Update: Same for `vps\d+.pairvpn.com/~monkads/?` and other mirror domains (???) In the templates for `displaytype=print` etc Add a meta tag for robots disallow I guess that should do it AFAICS are there also possibilities for settings in `.htaccess` for non-html file-types. YMMV Update If say the settings should be for all user agents. I suppose Google is using some helpful heuristics already. Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery}	[reply] [d/l] [select]

Replies are listed 'Best First'.

I'm still hoping/waiting for input from someone experienced.

From what I've read so far:

vps\d+.pairvpn.com/~monkads/?

AFAICS are there also possibilities for settings in .htaccess for non-html file-types.

YMMV

If say the settings should be for all user agents.

I suppose Google is using some helpful heuristics already.

Cheers Rolf
_{(addicted to the Perl Programming Language :)

Wikisyntax for the Monastery}