Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Need Fast LWP GET

by bobinyec (Novice)
on Jun 12, 2020 at 18:50 UTC ( #11117992=perlquestion: print w/replies, xml ) Need Help??

bobinyec has asked for the wisdom of the Perl Monks concerning the following question:

I'm using LWP to send a GET request to a server. Unfortunately, my code seems to wait until the server is finished executing the request before proceeding. Is there a way to just send a request to the server and keep moving REGARDLESS of what happens at the server?



Replies are listed 'Best First'.
Re: Need Fast LWP GET (updated)
by haukex (Bishop) on Jun 12, 2020 at 20:26 UTC

    Have a look at Corion's Future::HTTP, it supports multiple backends that allow for asynchronous HTTP requests. Just note that the default backend, if none of the event loops could be found, is HTTP::Tiny, which is synchronous, so make sure to load (and start) one of the supported event loops.

    Update: Now that Corion has released Future::HTTP 0.14, you can use the is_async method to confirm that the requests will indeed be asynchronous, e.g. my $ua = Future::HTTP->new(); die "UA is not asynchronous" unless $ua->is_async;

Re: Need Fast LWP GET
by hippo (Chancellor) on Jun 12, 2020 at 19:06 UTC

    There are several ways. As you've said nothing about your environment it's difficult to give specific advice. But you might consider

      That's the devil of it. The LWP request is in a fork, like this:
      fork If parent Do A If child Do B

      B has the LWP request in it. Branch A sends PRINT commands to a browser. The page being presented is held up until the LWP request in Branch B is finished.

      Maybe the fork command needs to be modified somehow so that both branches are truly independent.



Re: Need Fast LWP GET
by Marshall (Canon) on Jun 13, 2020 at 01:01 UTC
    A GET request is normally synchronous, meaning that it will wait for a response.

    Please explain what you mean by "keep moving REGARDLESS of what happens at the server".
    In general if you make a request like you want (an asynchronous request), you need to have code to check back upon what happened to that request while your code was doing something else. That can get complicated.

    There are multiple ways to create multiple instances of your program. Each instance sends a GET and waits. But there are multiple instances of this program running at once. So the server can have multiple requests that it is processing at the same time.

    Please explain further what you are intending to do?
    Sometimes there are API's that can do what you want.
    Sometimes there are "use policies" that restrict how many "get's" are outstanding at a given time for a particular website.

    Your general question is so vague that I have no solutions.

Re: Need Fast LWP GET
by perlfan (Vicar) on Jun 13, 2020 at 04:17 UTC
    This is generally handled by using fork. But if you don't watch how many children you're creating, you will fork bomb your system. I've found Parallel::ForkManager is a really nice, unobstrusive layer above fork that provides pretty much what you're going to need. That said, there are a variety of async modules out there; but I am pretty sure they all fork under the hood since Perl doesn't support threads unless they've written some fancy pthreaded XS backends (which maybe some have).
      Thank you to all who responded.

      Manipulating STDOUT and STDERR with CLOSE/OPEN didn't work.

      I'm trying to do this on a commercial web hosting service so I don't have access to the more non-mainstream solutions mentioned.

      Fortunately, the target process of the LWP isn't intolerably long, so the situation is acceptable.


        I think you also have to close STDIN, perhaps depending on which web server is being used.

        I'm trying to do this on a commercial web hosting service so I don't have access to the more non-mainstream solutions mentioned.

        Please see Yes, even you can use CPAN - if you have shell access to the machine, it's very likely you can install modules there, using something like local::lib, and sometimes it's possible even if you don't have shell access. Personally I'd prefer to help you with that rather than some other hacked solution :-)

        There is perlbrew. Also a VM on digital ocean is $5/mo, which provides a nice companion to any share hosting account environment. Those are not suitable for the kind of shell home you want. Also check out SDF/freeshell.
Re: Need Fast LWP GET
by karlgoethebier (Abbot) on Jun 14, 2020 at 09:47 UTC

    Probably helpful or inspiring.

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11117992]
Approved by Paladin
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2020-10-23 02:44 GMT
Find Nodes?
    Voting Booth?
    My favourite web site is:

    Results (234 votes). Check out past polls.