comment on

Hi, I need to work out a way to read log files from latest entry to earliest (so in reverse) quickly and with minimum overhead (so reading line by line rather than loading the whole file into memory). When I've benchmarked this, reading a file line by line is vastly quicker than reading the log file into memory (some of these log files are very large). I also have to use what is installed on these machines (of which there are thousands of them I need to run it on), so that rules out using File::ReadBackwards. So I opted to use a system command 'tac' to read the file and output it to a filehandle:

    
    my @loglist = nsort(@oldmsglogs);
    push(@loglist, "$basemsglog");
    my @sortedlogs = (reverse @loglist);
    for $msglogfile (@sortedlogs) {
      if ($done eq 0) {
        $logfile = join('/', $baselogdir, $msglogfile);
        open(RDMESG, "-|", "tac", $logfile);
        while ( <RDMESG> ) {
          my $line = $_;
          if ($line =~ m/(^[A-Z][a-z][a-z]\s+\d{1,2}\s+\d{1,2}\:\d{1,2
+}\:\d{1,2})(.*)/) {
            my $logtime = $1;
            my $msg = $2;
            my $logutime = str2time("$logtime");
            if ($logutime ge $unixstarttime) {
              if ($msg =~ m/kernel\:\s+CPU\d{1,3}\:\s+Package\s+temper
+ature\s+above\s+threshold\,\s+cpu\s+clock\s+throttled/) {
                $ctcnt++;
              }
            } else {
              $done = 1;
              last;
            }
          }
        }
        close RDMESG;
      } else {
        if ($ctcnt > $threshold) {
          $ret = sprintf("$warnmsg %d", $ctcnt);
          return $ret;
        } else {
          return $ret;
        }
      }
    }
    if ($ctcnt > $threshold) {
      $ret = sprintf("$warnmsg %d", $ctcnt);
      return $ret;
    }
[download]

However, I then have an issue with a broken pipe ('tac: write error: broken pipe'), since I close the filehandle while 'tac' is still writing to it with 'last', and this I do so that I stop reading the file once it falls out of the date range I'm interested in. If I take out 'last', then the error goes away, but it ends up reading the file to the end (or actually start of the file, since it is in reverse), which slows it down enormously, making it not viable. Is there a better way to do this, given the constraints I've mentioned? Perhaps a different way to invoke tac or an elegant way to terminate tac when I want to close the filehandle? Or some other way of doing this entirely? Any help appreciated! Cam

In reply to efficient way to read a file in reverse by cmcl

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Do you know where your variables are?
	PerlMonks