LWP::UserAgent lengthy download

jonnyfolk has asked for the wisdom of the Perl Monks concerning the following question:

I'm using LWP::UserAgent to interact with a third party search engine. The full list of searchable items is about 1500. If I restrict the search to a small number of results the delay is manageable. However the more items the more lengthy the delay.

If I go to the page in my browser the search is actually much more rapid, the results are printed but each result has an image and it takes a long time for the page to complete loading.

I'm wondering if the delay on printing the results via LWP is because the server waits for activity to stop before it prints the results. In which case is it possible to stop the script prematurely and print any output to screen - I could then try to balance the times to achieve a happy medium.

Or if there are other things that might be tried...

Comment on LWP::UserAgent lengthy download

Replies are listed 'Best First'.
Re: LWP::UserAgent lengthy download by BrowserUk (Patriarch) on Feb 17, 2010 at 07:19 UTC
. If I restrict the search to a small number of results the delay is manageable. However the more items the more lengthy the delay. If you can restrict the search, can you not split the required search into several restricted searches and do them serially? That'd allow you to display the intermediate results as you get them, and so reduce the pregnant pause up front. Possibly at the cost of taking slightly longer overall. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "I'd rather go naked than blow up my ass"	[reply]
Re^2: LWP::UserAgent lengthy download by jonnyfolk (Vicar) on Feb 17, 2010 at 08:18 UTC
That's an interesting idea, thank you. How would you display further results? One way might be to print the first split to screen, and the rest to a file, putting a link on the screen results. Did you have something else in mind?	[reply]
Re^3: LWP::UserAgent lengthy download by BrowserUk (Patriarch) on Feb 17, 2010 at 08:31 UTC
I assume from your question that you are representing the data you're fetching with LWP to a browser. I've made substantial use of LWP, but done very little with cgi, so I'm not the right person to answer that question. I have vague memories of something called multi-part data--and wikipedia kicks up this which may be related--but I know little about it. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "I'd rather go naked than blow up my ass"	[reply]
Re: LWP::UserAgent lengthy download by Anonymous Monk on Feb 17, 2010 at 07:08 UTC
Browsers have this ability to avoid downloading images ....	[reply]
Re^2: LWP::UserAgent lengthy download by jonnyfolk (Vicar) on Feb 17, 2010 at 10:18 UTC
But is this available in LWP::UserAgent - I can't find reference to it.	[reply]
Re^3: LWP::UserAgent lengthy download by Corion (Patriarch) on Feb 17, 2010 at 10:22 UTC
If you don't make an explicit request for an image (URL), LWP::UserAgent won't download the image. That's how HTTP works.	[reply]
Re^4: LWP::UserAgent lengthy download by jonnyfolk (Vicar) on Feb 20, 2010 at 14:21 UTC
Re^5: LWP::UserAgent lengthy download by Corion (Patriarch) on Feb 20, 2010 at 14:51 UTC
Some notes below your chosen depth have not been shown here


Clear questions and runnable code get the best and fastest answer
	PerlMonks