Dave.

Well, is there a way to set the buffer to a large size?

4K (or whatever buffer size perl is using) is likely to be a pretty good size for an input buffer, based on lots of experience and tweaking among perl maintainers. It strikes a nice balance between competing resource demands -- a larger or smaller size might improve some things, but hinder others.

Your processing is going to be line-oriented anyway, and perl's internal buffering is already optimized (in C) to deliver lines while managing the underlying block-oriented buffering.

If you try doing the buffering yourself (e.g. using read as suggested in another reply), you'll end up slowing things down, because you have to write your own code to figure out the line boundaries, retain a buffer-final line fragment so that you can append the next buffer to that, and so on. It's not only slower to run, it's slower and harder to code, test and maintain.

If the runtime speed of the standard while (<>) loop in perl is a serious issue for your task, maybe you just need to use C. But then you'll spend even more time for coding, testing and maintenance. It's a question of whose time is more important and expensive: the programmer's, or the cpu's.

You assume incorrectly. Perl will use the underlying IO routines (stdio or PerlIO or what not) to read chunks of the file in and will return just up to the next record separator at a time. Until this behind the scenes buffer is emptied (presuming the chunk read was large enough to contain up until the subsequent record separator) no disk IO will occur.

I'm not a Perl guru, but I have been knocking around computing for several decades. My experience has been that I/O buffering, especially read i/o buffering, is managed by the o/s, via the drivers. Again, from my experience, there is fairly little you can do at the application level to manage this.

You could, in theory set $/ to a (fairly large) numerical value, and see if it makes a noticeable difference, but I suspect the overhead introduced in explicitly processing end-of-record markers would eat up any savings.

-sam

   while ( read($fh, $buf, $len) ) {
       @lines = split "\n", $buf;
       .
       .
       .
   }
[download]

The line by line approach has a single loop with all the round trip times (from disk to destination). Cutting that into multiple processes/threads makes better use of resources.

buffer < infile | your_app.pl
[download]

Lots of games are being played here. The OS (if it is Linux, at least) may detect that you're doing a sequential read of the file and start doing read-ahead to get the data you're likely to read into kernel RAM before you even issue the read() for the next bit.

Your hard drive has a few (8? 16?) MBytes of RAM on it and again does similar readahead tricks.

Basically, everyone has optimised everything for the common case of the application developer sequentially reading a file from start to finish, so just go for it :-)

If you're still interested in tweaking, check out iostat, vmstat and sar to profile your running system to see what resource is being maxed. If you hit 100% disk utilisation, then you might be disk limited. In which case you can compare the time taken for a run of your app against the time taken by 'dd if=/your/file of=/dev/zero bs=4096', which should be pretty much a best-case for your box. You can even play with different chunk sizes with that command if you want to see if that makes a noticeable difference. Or create some soft RAID arrays if you have multiple disks and too much time on your hands.

Oh, and if you are going to take timings like that, you'll have to reboot in between each one and not read the file, otherwise you won't be reading it from disk - some of it may have been cached.