Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Parallel::ForkManager right approach

by davido (Cardinal)
on Oct 14, 2019 at 05:09 UTC ( [id://11107405]=note: print w/replies, xml ) Need Help??


in reply to Parallel::ForkManager right approach

Are forks needed for this, or could you be making your API requests using a non-blocking user agent?

Forks have their own overhead, and when you make an HTTP request against an API, most of your time is spent just sitting around waiting for the response -- you're not CPU bound. If this describes your situation, have a look at Mojo::UserAgent, Mojo::IOLoop, and Mojo::Promise. There are other options: AnyEvent::UserAgent, for example, but Mojo::UserAgent makes such tidy work of the problem, it's a really nice option.

Of course I don't know what type of API you're hitting. Hopefully this suggestion is useful in your case. Example:

my @promises = map {$ua->get_p($_)} @api_points; Mojo::Promise->all(@promises)->then(sub { my (@results) = @_; print Dumper $_->[0]->{'result'}->json foreach @results; });

If you have six @api_points this will make six GET requests in rapid fire and then process the results when they all return. There's a little code that I didn't show that surrounds this; using Mojo::UserAgent, getting a $ua object, kicking off the event loop if it hasn't started, that sort of thing. But it's all documented in the links above.


Dave

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11107405]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chanting in the Monastery: (7)
As of 2024-04-25 16:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found