Delayed writes

SpaceAce has asked for the wisdom of the Perl Monks concerning the following question:

I feel boneheaded asking this, because I am probably overlooking the obvious, but it's bugging me.

I have written a script that runs from a webserver and is controlled/executed through the browser. Because of the length of time that can be involved in running the script from beginning to end, it forks off and sends a simple "The program is running" message to the user. As the script is running, it logs its actions to a file that can be viewed from the script's control panel. The idea is that the user can check the log every once in a while to monitor progress and see when the script has completed execution. The problem is, the log is not being written in anything even approaching realtime. All the writing seems to be getting done in 4096 byte increments. When I have the script output text directly to the browser, I experience no such delays.

All I can think of is $|, but I have that set to a positive integer (1). I am not flocking the file (I had been, but both flock, 1 and flock, 2 resulted in the user having to wait for the script to run its course before viewing the log.

Right now, I am dancing around the problem by having the program close and re-open the log file at the beginning of each loop iteration but that seems like a silly way to go.

What's really driving me crazy is I used a nearly identical fork/log setup for a program in the past and it worked fine. Now I can't find that code anywhere, so I can't compare it with my current project. It's this part that leads me to believe I am overlooking some glaringly obvious point.

The problem is hardly fatal, but I hate not understanding what my program is doing. Any information on how to fix this would be appreciated.

Thanks, SpaceAce

Comment on Delayed writes

Replies are listed 'Best First'.
Re: Delayed writes by Helter (Chaplain) on Oct 28, 2002 at 18:13 UTC
You don't show code...but a suggestion, setting $\| only works on the currently selected file handle, so make sure you are doing something like: `select( FILEHANDLEFORMYFILE ) $\| = 1; select( STDOUT ); # I always like to set things back to how they were +before` [download] Hope this helps!	[reply] [d/l]
Re: Re: Delayed writes by fruiture (Curate) on Oct 28, 2002 at 18:23 UTC
I always like to set things back to how they were Then the following should be much better, because it really resets the selected handle, no matter which was selected. `my $oh = select( WHATEVER ); $\| = 1; select($oh); # or even (might be regarded as obfu) : select( ( select( WHATEVER ), $\| = 1 )[0] );` [download] -- http://fruiture.de	[reply] [d/l]
Re: Re: Delayed writes by SpaceAce (Beadle) on Oct 28, 2002 at 18:50 UTC
Ooooooh, that might be the glaringly obvious thing I am missing :) (Note: yep, it was) It's just like that feeling you get when you leave the house and you just KNOW you forgot something... Just for the record, I didn't post any code because the workhorse function where this is going on is a couple of hundred lines long and I was hoping to get a general suggestion like yours. Thanks for the help. SpaceAce	[reply]
Re: Delayed writes by nothingmuch (Priest) on Oct 28, 2002 at 18:27 UTC
Since a fix was given, now an explenation: Because the STDIO buffers are set to 4k in your libraries, whenever perl prints/writes/printfs, the data you were going to print is stored in memory. When the buffer in the memory reaches 4k, it is 'flushed', meaning a system call is made, to output the data to disk. This is in order to save on somewhat more expensive system calls. the $\| variable is the correct solution, as it forces a buffer flush after every such STDIO output command. It is, however, sensitive to the currently selected filehandle. What you have to do in order to apply it to the file's filehandle, instead of to STDOUT, is mentioned in the previous reply. Setting $\| only applies to the currently selected handle, and when you select another handle, the value still remains. A different way to handle this is to make the system call explicitly, replacing all outputs with syswrite, and all inputs with sysread (Mixing the STDIO (all other) I/O commands with the ones prefixed with 'sys' will tend to lead to confusion, as parts of the file are in memory. When an STDIO write is performed, writing is postponed until the buffer is full, and when an STDIO read is performed, the buffer is filled. This will cause syswrites to occasionally output before previous STDIO outs, and sysreads to skip bytes forward, when mixed with STDIO reads. Keep them seperate - the documentation has more on this). This will work around the buffers completely, and will have the same effect as flushing them each time around. Hope this helps! -nuffin zz zZ Z Z #!perl	[reply]


Perl: the Markov chain saw
	PerlMonks