http://qs321.pair.com?node_id=517718


in reply to Advice on Efficient Large-scale Web Crawling

Personaly, I think you're engaging in premature optimization here: when fetching 4M urls, the DNS traffic is unlikely to be your biggest concern.

Having said that, the cheapest/cleanest method would be to install a caching-only DNS server on your localhost, and let it handle the DNS caching.

Some reasons why your current solution might be slow: