|
|
|
| Think about Loose Coupling | |
| PerlMonks |
Re^5: Web Robotby schumi (Hermit) |
| on Jul 17, 2003 at 09:45 UTC ( [id://275160]=note: print w/replies, xml ) | Need Help?? |
|
Quite true, although most major search engines do actually heed the robots-file, if it is setup properly.
I think the easiest way to restrict access to a directory is setting up a proper .htaccess-file. You could even restrict access by IP-addresses... On the other hand, using a robots-file (in addition to the above, note!) decreases the amount of 404s in your error-logs... ;-) --cs There are nights when the wolves are silent and only the moon howls. - George Carlin
In Section
Meditations
|
|
||||||||||||||||||||||