in reply to Re: Re: Web Robot
in thread Web Robot
Most major search engines don't stick to those rules, true. That means that if you use a robots.txt on your site, be aware of that.
But I think, just because some big companies/search engines don't stick to the rules doesn't mean that you should do the same. I always go by the maxime, don't do unto someone else, what you wouldn't want done to you/your site.
Just my 2 Rappen (Swiss equivalent to cents).
--cs
There are nights when the wolves are silent and only the moon howls. - George Carlin
In Section
Meditations