http://qs321.pair.com?node_id=275152


in reply to Re: Re: Web Robot
in thread Web Robot

Most major search engines don't stick to those rules, true. That means that if you use a robots.txt on your site, be aware of that.

But I think, just because some big companies/search engines don't stick to the rules doesn't mean that you should do the same. I always go by the maxime, don't do unto someone else, what you wouldn't want done to you/your site.

Just my 2 Rappen (Swiss equivalent to cents).

--cs

There are nights when the wolves are silent and only the moon howls. - George Carlin