Re: perl regex or module that identifies bots/crawlers


Problems? Is your data what you think it is?
	PerlMonks

Re: perl regex or module that identifies bots/crawlers

by sgifford (Prior)

on Mar 20, 2007 at 21:15 UTC ( [id://605750]=note: print w/replies, xml )

Need Help??

in reply to perl regex or module that identifies bots/crawlers

Google and Yahoo should certainly be honoring your robots.txt file. You might want to take a closer look, to see what IP address these requests are coming from and what URLs they are fetching; perhaps there is another path to your cgi-bin directory that isn't being protected by your robots file, or maybe there is an error that's preventing your robots file from being processed correctly.

--
sgifford's Web page

Comment on Re: perl regex or module that identifies bots/crawlers Select or Download Code

Replies are listed 'Best First'.
Re^2: perl regex or module that identifies bots/crawlers by Anno (Deacon) on Mar 20, 2007 at 22:22 UTC
I agree that the real Google and Yahoo, and other big ones, will certainly honor `robots.txt`. If bots under their names invade a server that may only indicate that these are popular fake names for rogue bots. It would make sense to look like a legit bot instead of, for instance, a browser. That said, it is certainly a good idea to check if `robots.txt` is working as it should. Anno	[reply] [d/l] [select]

In Section Seekers of Perl Wisdom

Domain Nodelet^?

www.com | www.net | www.org

Node Status^?

node history
Node Type: note [id://605750]
help

Chatterbox^?

How do I use this? • Last hour • Other CB clients

Other Users^?

Others romping around the Monastery: (9)

As of 2024-04-18 11:09 GMT

Sections^?

Information^?

Find Nodes^?

Leftovers^?

Today I Learned

Voting Booth^?

No recent polls found