request timing out on Perl Web

jpcardoso has asked for the wisdom of the Perl Monks concerning the following question:

Hi guys,

I'm not sure if this is a Perl question, but I'm developing a website in Perl and it contains a line that calls for a grep in a bzcat of a .bz2 file.

It's something like this:

my $var = `bzcat $path/file.bz2 | grep -i word`;
print "$var";
[download]

The issue is that when I load the perl in the web browser, the page stops loading before the bzcat ends. The bzcat is taking around 5 minutes to complete but the browser "cuts" the connection and the page does not load the content.

Running the perl in the terminal it prints the content without problems.

My question is: Is there a way to increase the timeout of the bzcat process running on web?

Thanks in advance.

Comment on request timing out on Perl Web Download Code

Replies are listed 'Best First'.
Re: request timing out on Perl Web by flexvault (Monsignor) on May 30, 2014 at 15:18 UTC
Welcome jpcardoso, What is the error message in the web server log? Usually the default is around 5 minutes, but the parameter is in seconds, so you could just be missing it. Regards...Ed "Well done is better than well said." - Benjamin Franklin	[reply]
Re^2: request timing out on Perl Web by jpcardoso (Initiate) on May 30, 2014 at 16:07 UTC
Hi Ed, The error message in the webserver log is "Timeout waiting for output from CGI script". You opened my mind and I set the TimeOut value to 360 in the apache configuration, I was missing that. :) It seems to be working fine now, thank you very much for the quick help!	[reply]
Re^3: request timing out on Perl Web by flexvault (Monsignor) on May 30, 2014 at 17:46 UTC
jpcardoso, I'm glad I could help you, but I also have to agree with the other monks about waiting 5 minutes for a response. That's a really long time on the web today. Most users expect responses in seconds, not minutes. Doing this lookup with 'ajax' would be nice, but it would also be a lot of work. Maybe, let us help you with the real problem -- having to 'grep' on a 'bz2' file. Perl is very good at reading a file and searching ( 'index' ). Several generic fixes come to mind, but some more information about the data(bz2 file) would help us give you some better options. But just in case you can't get around the 'grep', you could schedule it and immediately return a HTML page informing the user that it will take 'x' minutes to get a response and to check back in a few minutes. Provide a link to check on the status. Once the response is available, return the current page that you are returning now. You may also want to look at the problem of denial of service (DOS) with longer time-outs. The apache web site describes this in detail. Better to plan now, than get the web site finished and then have a DOS attack. Best of Luck...Ed "Well done is better than well said." - Benjamin Franklin	[reply]
Re: request timing out on Perl Web by choroba (Cardinal) on May 30, 2014 at 15:45 UTC
Similar questions have been already posted and answered. Have you tried to search the site? Search for: CGI long running لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ	[reply]
Re^2: request timing out on Perl Web # preloaded supersearch by LanX (Saint) on May 30, 2014 at 16:09 UTC
unfortunately newbies always need to be informed to click on search after following the link. (Would be nice if `Link to preload this search:` would automatically produce a red search button on top with bold text telling people to click ) Cheers Rolf ( addicted to the Perl Programming Language)	[reply] [d/l]
Re^2: request timing out on Perl Web by jpcardoso (Initiate) on May 30, 2014 at 16:15 UTC
Actually I searched, but probably used the wrong words. Anyway, thanks for the one who helped me.	[reply]
Re: request timing out on Perl Web by taint (Chaplain) on May 30, 2014 at 17:14 UTC
My first thought reading this was to answer suggesting some `while {...}`, emitting some form of progress information -- a scale, or a repeating "Please wait..." output, ... This would have served 2 purposes; 1) overcome "timeout thresholds"; both the web server, as well as the web client (browser). 2) provide feedback to the user; the threshold of users attention span has greatly diminished, since the dawn of the internet. All the best. --Chris ¡λɐp ʇɑəɹ⅁ ɐ əʌɐɥ puɐ ʻꜱdləɥ ꜱᴉɥʇ ədoH	[reply]
Re: request timing out on Perl Web by Anonymous Monk on May 31, 2014 at 00:22 UTC
See Watching long processes through CGI (Aug 02) and Proc::Background	[reply]
Re: request timing out on Perl Web by sundialsvc4 (Abbot) on May 31, 2014 at 01:47 UTC
I usually make the suggestion that “a web-page should be thought of as a user-interface, nothing more.” If there is “more” to be done, then the web-page interface ought to be used to initiate it, to monitor it, and to retrieve the results from it (whenever the user cares to stop by again and (s)he sees that the work is now complete), but the web page should never be the one to actually do it. Some other background-processing engine, and there are a great many to choose from, should be doing that part of it. Ideally, everything that a single web-page HTTP transaction should be asked to do, should take only a few milliseconds between request and response. Even if the user checks back hours later or the next day, the results should be there waiting to be picked up.
Re: request timing out on Perl Web by Laurent_R (Canon) on May 30, 2014 at 17:10 UTC
I might have misunderstood, but do you seriously expect your user to wait more that 5 minutes for your page to be displayed?	[reply]

Back to Seekers of Perl Wisdom