Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Searching the monastery with duckduckgo leads to ugly results

by Corion (Patriarch)
on Nov 01, 2021 at 13:58 UTC ( [id://11138294]=note: print w/replies, xml ) Need Help??


in reply to Searching the monastery with duckduckgo leads to ugly results

I don't really understand the findings.

Is there anything actionable here? Should we block certain? known? bots from visiting certain pages?

  • Comment on Re: Searching the monastery with duckduckgo leads to ugly results

Replies are listed 'Best First'.
Re^2: Searching the monastery with duckduckgo leads to ugly results
by LanX (Saint) on Nov 01, 2021 at 14:13 UTC
    Hi

    I'm still hoping/waiting for input from someone experienced.

    From what I've read so far:

    On www.perlmonks.org (the main target)
    • disallow /bare/
    • disallow /mobile/
    On qs\d+.pair.com domains
    • disallow /
    Or
    • disallow /~perl2/
    (Update: Same for vps\d+.pairvpn.com/~monkads/? and other mirror domains (???)

    In the templates for displaytype=print etc
    • Add a meta tag for robots disallow
    I guess that should do it

    AFAICS are there also possibilities for settings in .htaccess for non-html file-types.

    YMMV

    Update

    If say the settings should be for all user agents.

    I suppose Google is using some helpful heuristics already.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

[reply]
[d/l]
[select]

In Section Perl Monks Discussion

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://11138294]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2024-04-24 22:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found