what the force of many cannot
|
|
|
|
Your skill will accomplish what the force of many cannot |
|
| PerlMonks |
Re: What is the fastest way to download a bunch of web pages?by lestrrat (Deacon) |
| on Mar 03, 2005 at 16:08 UTC ( [id://436260]=note: print w/replies, xml ) | Need Help?? |
|
Just to make things more interesting, I'd suggest you take a look at even based approach, for example, via POE (POE::Component::Client::HTTP) or the like. But I'd suggest that you keep this in the back of your head, and leave it for future, because it requires that you think about I/O, order of things, blah blah blah. It was pretty hard for me personally to write a web crawler like that. But anyway, it *is* possible to increase the performance of fetching websites to about 10K ~ 20K urls/hour using such an approach. And this is with a single process.
In Section
Seekers of Perl Wisdom
|
|
||||||||||||||||||||||||||||||