Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Increasing the write buffer

by accassar (Initiate)
on Jun 21, 2008 at 08:22 UTC ( [id://693259]=note: print w/replies, xml ) Need Help??


in reply to Increasing the write buffer

Opening 30K+ files is not a problem. I was hoping for some hidden mechanism to better control the output buffering - however your idea of manually controlling it is not a bad one. I heard that when writing "\n" to an output stream it forces a flush? Can anybody confirm this?

Replies are listed 'Best First'.
Re^2: Increasing the write buffer
by ww (Archbishop) on Jun 21, 2008 at 10:23 UTC
      And just as a matter of curiousity, what OS so generously gives you 30K + X simultaneously open file_handles?

      Linux probably does. I've never actually tried it, but if I cat /proc/sys/fs/file-max, I get a number like 300,000. I'm assuming the usual lower limits are probably set by the shell on login... ulimit -n shows 1024 for me.

      I imagine, after a certain point, having that many open file handles is counter productive, but I could be wrong.

      -Paul

Re^2: Increasing the write buffer
by ikegami (Patriarch) on Jun 21, 2008 at 17:54 UTC

    There are three buffering states:

    • unbuffered
    • block buffered: Writes when 4KB* is accumulated.
    • line buffered: Writes when \n is encountered or 4KB* is accumulated.

    If a handle is buffered, line buffering is always and only used when the file handle is connected to a terminal.

    So, if you're writing to a file, \n won't flush. But if you're writing to STDOUT and it hasn't been redirected, \n will flush if buffering wasn't turned off.

    You can turn on buffering using

    use IO::Handle qw( ); FH->autoflush(0);

    You can turn off buffering using

    use IO::Handle qw( ); FH->autoflush(1);

    You can flush manually using

    use IO::Handle qw( ); FH->flush();

    * — Well, I *think* the buffer size is 4KB.

Re^2: Increasing the write buffer
by starbolin (Hermit) on Jun 22, 2008 at 01:32 UTC

    Writing a "\n" to the disk does not force a flush. From perlvar:

    STDOUT will typically be line buffered if output is to the terminal and block buffered otherwise.
    Autoflush only applies if select() has been called to connect the handle to STDOUT and STDOUT is directed to the terminal.

    In general everything you write to the disk is greedily buffered by the OS unless and until the OS runs out of buffer-cache, at which point everything slows to a crawl.


    s//----->\t/;$~="JAPH";s//\r<$~~/;{s|~$~-|-~$~|||s |-$~~|$~~-|||s,<$~~,<~$~,,s,~$~>,$~~>,, $|=1,select$,,$,,$,,1e-1;print;redo}

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://693259]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others browsing the Monastery: (None)
    As of 2024-04-25 00:35 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      No recent polls found