Yeah, I saw that one, too. It was actually a one-liner: A Web Spider in One Line?

The article also mentions a good source of information about spiders: The Web Robots Pages.

-jehuni