Re: Need Fast LWP GET (updated)
by haukex (Archbishop) on Jun 12, 2020 at 20:26 UTC
|
Have a look at Corion's Future::HTTP, it supports multiple backends that allow for asynchronous HTTP requests. Just note that the default backend, if none of the event loops could be found, is HTTP::Tiny, which is synchronous, so make sure to load (and start) one of the supported event loops.
Update: Now that Corion has released Future::HTTP 0.14, you can use the is_async method to confirm that the requests will indeed be asynchronous, e.g. my $ua = Future::HTTP->new(); die "UA is not asynchronous" unless $ua->is_async;
| [reply] [d/l] [select] |
Re: Need Fast LWP GET
by hippo (Bishop) on Jun 12, 2020 at 19:06 UTC
|
There are several ways. As you've said nothing about your environment it's difficult to give specific advice. But you might consider
| [reply] |
|
That's the devil of it. The LWP request is in a fork, like this:
fork
If parent
Do A
If child
Do B
B has the LWP request in it. Branch A sends PRINT commands to a browser. The page being presented is held up until the LWP request in Branch B is finished.
Maybe the fork command needs to be modified somehow so that both branches are truly independent.
Thanks,
jb
| [reply] [d/l] |
|
| [reply] |
Re: Need Fast LWP GET
by Marshall (Canon) on Jun 13, 2020 at 01:01 UTC
|
A GET request is normally synchronous, meaning that it will wait for a response.
Please explain what you mean by "keep moving REGARDLESS of what happens at the server".
In general if you make a request like you want (an asynchronous request), you need to have code to check back upon what happened to that request while your code was doing something else. That can get complicated.
There are multiple ways to create multiple instances of your program. Each instance sends a GET and waits. But there are multiple instances of this program running at once. So the server can have multiple requests that it is processing at the same time.
Please explain further what you are intending to do?
Sometimes there are API's that can do what you want.
Sometimes there are "use policies" that restrict how many "get's" are outstanding at a given time for a particular website.
Your general question is so vague that I have no solutions.
| [reply] |
Re: Need Fast LWP GET
by perlfan (Vicar) on Jun 13, 2020 at 04:17 UTC
|
This is generally handled by using fork. But if you don't watch how many children you're creating, you will fork bomb your system. I've found Parallel::ForkManager is a really nice, unobstrusive layer above fork that provides pretty much what you're going to need. That said, there are a variety of async modules out there; but I am pretty sure they all fork under the hood since Perl doesn't support threads unless they've written some fancy pthreaded XS backends (which maybe some have).
| [reply] |
|
Thank you to all who responded.
Manipulating STDOUT and STDERR with CLOSE/OPEN didn't work.
I'm trying to do this on a commercial web hosting service so I don't have access to the more non-mainstream solutions mentioned.
Fortunately, the target process of the LWP isn't intolerably long, so the situation is acceptable.
jb
| [reply] |
|
| [reply] |
|
|
I'm trying to do this on a commercial web hosting service so I don't have access to the more non-mainstream solutions mentioned.
Please see Yes, even you can use CPAN - if you have shell access to the machine, it's very likely you can install modules there, using something like local::lib, and sometimes it's possible even if you don't have shell access. Personally I'd prefer to help you with that rather than some other hacked solution :-)
| [reply] |
|
There is perlbrew. Also a VM on digital ocean is $5/mo, which provides a nice companion to any share hosting account environment. Those are not suitable for the kind of shell home you want. Also check out SDF/freeshell.
| [reply] |
Re: Need Fast LWP GET
by karlgoethebier (Abbot) on Jun 14, 2020 at 09:47 UTC
|
| [reply] [d/l] |