IPC::Run::run and buffering

rovf has asked for the wisdom of the Perl Monks concerning the following question:

I have an application, which uses IPC::Run in the following way:

IPC::Run::run([$cmd,@args],'>>',$outpath,'2>&1')
[download]

I have no information about $cmd - can be anything, and it is not on my control. Now my question:

Under the assumption that $cmd does not do unbuffered output, and doesn't fiddle around with buffering by itself (think of: just doing C style fwrite calls), is there a way I can control the buffer size (i.e. make it larger) of the pipe from the Perl side?

The code is currently executed on Windows and Linux. I understand that if there is at all a solution to my problem, it probably will look different on both platforms. This is OK. I'm not very optimistic, that a solution to this problem exist, but I don't know enough about the operating system internals.

Another idea which came to my mind, would be to replace the "IPC::Run::run" by an open(my $pipe,"$cmd ...|") and read from this pipe to my application, which then writes the output in larger chunks. This should work at least on Linux (possibly not Windows, because Windows does not, AFIK, support real pipes and needs to simulate them using files). However, if there were a solution of just increasing the buffer size for the output redirection, I would prefer that.

UPDATE:

I would like to sketch a possible solution - maybe someone can see whether or not it should work: I replace the call to IPC::Run::run by a call to IPC::Run::start and collect the output via a function:

my $harness=start(\@cmd,\'',\&f,\&f)
[download]

The function buffers the resulting lines, and if the number of characters exceeds a certain amount, it "flushes" the buffer by writing it to the output file. In order to bypass system-internal buffering, the output file is opened for writing using the :unix layer, and the data is written using syswrite instead of print. This is done in order to bypass stdio buffering. Of course this means that on Windows, I have to manually translate \n into 0x0d.0x0a manually.

Does this sound reasonable? Is there a simpler way to do it?

--
Ronald Fischer <ynnor@mm.st>

Comment on IPC::Run::run and buffering Select or Download Code

Replies are listed 'Best First'.
Re: IPC::Run::run and buffering by zentara (Archbishop) on Jun 14, 2012 at 10:58 UTC
I don't know why you need to increase buffer size, but see Configurable IO buffersize?. Just in case you feel you are getting data left over in the buffer, see IPC3 buffer limit problem. It isn't perfect ( as Merlyn pointed out ), but it basically works for Linux. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]
Re^2: IPC::Run::run and buffering by rovf (Priest) on Jun 14, 2012 at 13:06 UTC
I don't know why you need to increase buffer size It happens that several copies of this program are running simultaneously, and the all write to files on the same host. With the default buffer size, this causes too many I/O accesses, sometimes causing long delays until the actual writing operation has been completed. I would like to try weather increasing the buffer size, hence decreasing the number of simultaneous writes, would give a better performance. I checked the link you have provided, but it doesn't give much insight into my problem (i.e. Perl version greater than 5.8, and buffered writing (not reading)). BTW, I've updated my node. If you have time, it would be great if you could have a look at it again. -- Ronald Fischer <ynnor@mm.st>	[reply] [d/l]
Re: IPC::Run::run and buffering by Anonymous Monk on Jun 14, 2012 at 09:38 UTC
You're under the impression you need to increase the buffer size? Try to write a program that triggers whatever problem you think you're encountering, like this program `my $cmd = $^X; my @args = ( qw' -l -e ', 'print $_ for 1 .. 10_000;' );` [download]	[reply] [d/l]
Re: IPC::Run::run and buffering by BrowserUk (Patriarch) on Jun 14, 2012 at 14:39 UTC
See Re^6: Pipe Problem	[reply]


Syntactic Confectionery Delight
	PerlMonks