Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re^2: Fast fetching of HTML response code

by mrguy123 (Hermit)
on May 30, 2012 at 09:06 UTC ( [id://973234]=note: print w/replies, xml ) Need Help??


in reply to Re: Fast fetching of HTML response code
in thread Fast fetching of HTML response code

Hi Corion, thanks for the tip
I am currently using Parallel::ForkManager to make the process faster but will also take a look at AnyEvent::HTTP
The HEAD option, however, does not seem reduce to the time of response so I am looking for new ideas
Guy
  • Comment on Re^2: Fast fetching of HTML response code

Replies are listed 'Best First'.
Re^3: Fast fetching of HTML response code
by Corion (Patriarch) on May 30, 2012 at 09:09 UTC

    You likely can't reduce the time per URL, but you can reduce the total time by making more requests in parallel. Or you can try to find out whether you are bandwidth-limited, or CPU-limited or whether the remote side is slow to respond. My guess is that the remote side is slow to respond.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://973234]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (5)
As of 2024-04-25 05:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found