Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Re: LWP::Parallel vs. HTTP::GHTTP vs. IO::Socket

by mp (Deacon)
on May 16, 2003 at 16:42 UTC ( #258700=note: print w/replies, xml ) Need Help??


in reply to Re: LWP::Parallel vs. HTTP::GHTTP vs. IO::Socket
in thread LWP::Parallel vs. HTTP::GHTTP vs. IO::Socket

You can use Parallel::ForkManager to parallelize HTTP::MHTTP or HTTP::GHTTP calls easily and apply a limit to the maximum number of child processes.

There are a number of ways to handle getting the retrieved data back to the parent or other process that don't require use of shared memory:

  • Comment on Re: Re: LWP::Parallel vs. HTTP::GHTTP vs. IO::Socket

Replies are listed 'Best First'.
Re: LWP::Parallel vs. HTTP::GHTTP vs. IO::Socket
by hacker (Priest) on May 16, 2003 at 23:21 UTC
    As it turns out, HTTP::MHTTP seems to have an 'issue' with name-based virtual hosts, exhibited by the code below, so I can't use that, and it doesn't appear to work on Windows machines either, which puts it in the non-portable category for me:
    use strict; use HTTP::MHTTP; # This url REALLY exists, but is a virtual host # on a domain shared by multiple hosts. my $url = 'http://advogato.org/recentlog.html; http_init(); switch_debug(1); http_call("GET", $url); print http_response();

    Thanks to bart and ChemBoy for the enlightening discussion that exposed this issue.

    Based on my loose testing (excluding HTTP::MHTTP), it looks like HTTP::GHTTP is the fastest, followed closely by HTTP::Lite and LWP::Simple behind that. I haven't done benching against Parallel::ForkManager yet with these, so that waits to be seen.

    The other issue also, is the speed at which DNS queries are resolved. I think I can speed that up with a local database of resolved sites, but on the first run, that'll take a hit.

    Thanks for the tips and hints though, I'm closer to a functional solution, but it seems the more I test, the farther down the stack I get, closer to writing my own code around IO::Socket. I'd like to avoid that if I can.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://258700]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (2)
As of 2022-01-18 00:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (52 votes). Check out past polls.

    Notices?