http://qs321.pair.com?node_id=275160


in reply to Re: Re: Re: Re: Web Robot
in thread Web Robot

Quite true, although most major search engines do actually heed the robots-file, if it is setup properly.

I think the easiest way to restrict access to a directory is setting up a proper .htaccess-file. You could even restrict access by IP-addresses...

On the other hand, using a robots-file (in addition to the above, note!) decreases the amount of 404s in your error-logs... ;-)

--cs

There are nights when the wolves are silent and only the moon howls. - George Carlin