http://qs321.pair.com?node_id=1003855

anshumangoyal has asked for the wisdom of the Perl Monks concerning the following question:

There is a program which keeps on writing in a log file at a very fast rate. The rate is so fast that the file reached 250 MB in just 30 seconds. In perl script I want to write a program which rotates this file. i.e. when ever the size of this file reaches 100 MB, I want to flush all contents of this file and write in log_001.log so that all size remains 100 MB. Problem is when I rename the file, the file stops writing (may be due to file pointer being destroyed). What is the best possible thing that can be done here. I want the program to be efficient so that not too much of memory is being used as well.

Replies are listed 'Best First'.
Re: Log File Rotation
by fisher (Priest) on Nov 14, 2012 at 18:56 UTC
    The common approach to this problem is to
    1. rename log-file. Like mv file.log file.log.1. The process which holds the old file descriptor open will write to renamed file from this point.
    2. touch new log file. touch file.log or otherwise create it.
    3. inform the process. For example, send it SIGHUP or even restart it.
    It is not related to perl, I believe.
Re: Log File Rotation
by Tanktalus (Canon) on Nov 15, 2012 at 03:00 UTC

    Ideally, you get that program to do its own log rotation. Second choice is to get it to respond to a SIGHUP as mentioned earlier to close and reopen its log handles (you rename them yourself first, and then send the signal).

    Right near the bottom is to create a named pipe (see POSIX::mkfifo / mkfifo) on the log file location first, receiving all of the log entries, and you write them out to the rotating log files. It's moderately evil, but so is 500MB of logging per minute. You just sit there reading from the fifo pipe, and every line you get you can then pipe into Log::Log4perl / Log::Dispatch. You just have to configure these to use Log::Dispatch::FileRotate, and your problem is basically solved. :-) (Still evil, though.)

Re: Log File Rotation
by pvaldes (Chaplain) on Nov 14, 2012 at 23:49 UTC

    take a look to liblogfile-rotate-perl

    updated: the original module Rotate packed in liblogfile-rotate-perl, here

Re: Log File Rotation
by afoken (Chancellor) on Nov 15, 2012 at 07:08 UTC

    If you can make the program write its log to a file handle (STDOUT, /dev/fd/1), you can use multilog and a simple pipe:

    /usr/local/bin/your-program --logfile=/dev/fd/1 | /command/multilog t s100000000 /var/log/your-programs-logdirectory

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)