Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Speeding up/parallelizing hundreds of HEAD requests

by NetWallah (Canon)
on Sep 16, 2007 at 17:10 UTC ( [id://639273]=note: print w/replies, xml ) Need Help??


in reply to Speeding up/parallelizing hundreds of HEAD requests

This situation could be viewed as a cacheing issue, and addressed that way, without delving into the complexities of parallel processing.

If the source data is relatively static, and the requests are "browsing" style, then you potentially are retrieving the same data periodically, and cacheing would help.

The simplest would be to use the Memoize module. Minimally more work, but more accurate would be to use Memoize::Expire.

     "As you get older three things happen. The first is your memory goes, and I can't remember the other two... " - Sir Norman Wisdom

  • Comment on Re: Speeding up/parallelizing hundreds of HEAD requests

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://639273]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (3)
As of 2024-04-26 00:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found