comment on

This situation could be viewed as a cacheing issue, and addressed that way, without delving into the complexities of parallel processing.

If the source data is relatively static, and the requests are "browsing" style, then you potentially are retrieving the same data periodically, and cacheing would help.

The simplest would be to use the Memoize module. Minimally more work, but more accurate would be to use Memoize::Expire.

"As you get older three things happen. The first is your memory goes, and I can't remember the other two... " - Sir Norman Wisdom

In reply to Re: Speeding up/parallelizing hundreds of HEAD requests by NetWallah
in thread Speeding up/parallelizing hundreds of HEAD requests by hacker

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Keep It Simple, Stupid
	PerlMonks