Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Re: Re: Re: Web Robot

by Anonymous Monk
on Jul 17, 2003 at 09:10 UTC ( [id://275155]=note: print w/replies, xml ) Need Help??


in reply to Re: Re: Re: Web Robot
in thread Web Robot

On the topic of robots.txt, why would someone even use this? If you don't want a page accessed, limit access to it. Depending on all computers to play nice isn't a very smart move, they have many hidden motives :)

Replies are listed 'Best First'.
Re^5: Web Robot
by schumi (Hermit) on Jul 17, 2003 at 09:45 UTC
    Quite true, although most major search engines do actually heed the robots-file, if it is setup properly.

    I think the easiest way to restrict access to a directory is setting up a proper .htaccess-file. You could even restrict access by IP-addresses...

    On the other hand, using a robots-file (in addition to the above, note!) decreases the amount of 404s in your error-logs... ;-)

    --cs

    There are nights when the wolves are silent and only the moon howls. - George Carlin

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://275155]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2024-03-29 10:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found