|Welcome to the Monastery
Re^4: IPC::Open, Parallel::ForkManager, or Sockets::IO for parallelizing?by mldvx4 (Friar)
|on Sep 05, 2023 at 16:48 UTC
Have you tried just putting your processing in the handle_answer callback?
Yes, I had started to look at that. As far as I can tell handle_answer keeps getting called only until the HTTP response is complete. I suppose there is a way to identify when the response is finally complete? For now, I will try building out the script as-is, with LWP in parallel and the rest serial. Some of the preparatory parts seem much faster than expected, based on trials in Perl compared to another scripting language. So it may very well not be an issue. Although I have lots of RAM on this computer, I'd like to find a way to process the responses as they come in so that all 700 to 800 of them don't sit around whole at the same time.
Resuming more reading...