|
|
|
| No such thing as a small change | |
| PerlMonks |
Waiting my turn politely with LWPby toro (Beadle) |
| on Jun 17, 2011 at 08:00 UTC ( [id://910089]=perlquestion: print w/replies, xml ) | Need Help?? |
|
toro has asked for the wisdom of the Perl Monks concerning the following question: I read http://robotstxt.org and therefore know that I should identify my LWP::UserAgent and leave a contact email when I crawl robotically. I'd also like to avoid bombarding the servers I poke, especially during periods of high traffic for them. Currently I am sleeping between jobs and running cron jobs at boring American hours. Is there a more adaptive way (perhaps within LWP) to pause a crawl when a server says it's busy? And, anything else I should know to be a polite web crawler?
Back to
Seekers of Perl Wisdom
|
|
||||||||||||||||||||||||||||