Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re: Searching the monastery with duckduckgo leads to ugly results

by Corion (Patriarch)
on Nov 01, 2021 at 13:58 UTC ( #11138294=note: print w/replies, xml ) Need Help??


in reply to Searching the monastery with duckduckgo leads to ugly results

I don't really understand the findings.

Is there anything actionable here? Should we block certain? known? bots from visiting certain pages?

  • Comment on Re: Searching the monastery with duckduckgo leads to ugly results

Replies are listed 'Best First'.
Re^2: Searching the monastery with duckduckgo leads to ugly results
by LanX (Sage) on Nov 01, 2021 at 14:13 UTC
    Hi

    I'm still hoping/waiting for input from someone experienced.

    From what I've read so far:

    On www.perlmonks.org (the main target)
    • disallow /bare/
    • disallow /mobile/
    On qs\d+.pair.com domains
    • disallow /
    Or
    • disallow /~perl2/
    (Update: Same for vps\d+.pairvpn.com/~monkads/? and other mirror domains (???)

    In the templates for displaytype=print etc
    • Add a meta tag for robots disallow
    I guess that should do it

    AFAICS are there also possibilities for settings in .htaccess for non-html file-types.

    YMMV

    Update

    If say the settings should be for all user agents.

    I suppose Google is using some helpful heuristics already.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11138294]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (1)
As of 2022-01-17 05:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (51 votes). Check out past polls.

    Notices?