Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Reading constantly generated log files: Suggestions invited

by perlgsm (Initiate)
on Nov 08, 2013 at 17:49 UTC ( [id://1061736]=perlquestion: print w/replies, xml ) Need Help??

perlgsm has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,
A quick stupid question.

I need to build a script to read a directory for some logs files. There is one log file (almost) per day and gets written by some server, whenever there is something interesting.
I wrote a script getting input from 'tail -q -n 0 -f /path/file-2013-11-1-31|'.. Problem is since there is date in the file name (yyyy-mm-dd) so use of wild card gets the 'available' files for me to read.
Whenever a new file is available I don't get that, I verified this by doing ps and all I had was, the available file names with tail.
Question is how to get the new files available to be read as and when they appear in the directory?
I don't know Perl IPC, but I can explore that if needed.
Somehow I'm not getting any ideas for this tricky problem.

Ideas are invited!

Thank you in anticipation.

<! -->
God bless world peace...
  • Comment on Reading constantly generated log files: Suggestions invited

Replies are listed 'Best First'.
Re: Reading constantly generated log files: Suggestions invited
by Kenosis (Priest) on Nov 08, 2013 at 18:10 UTC

    As part of your complete solution, perhaps the module File::DirList would be helpful, as you can get the newest file in a directory from it:

    use strict; use warnings; use File::DirList; my $dir = '.'; my $newestFile = File::DirList::list( $dir, 'M' )->[0]->[13]; print $newestFile;

    Hope this helps!

Re: Reading constantly generated log files: Suggestions invited
by davido (Cardinal) on Nov 08, 2013 at 19:36 UTC
Re: Reading constantly generated log files: Suggestions invited
by Khen1950fx (Canon) on Nov 08, 2013 at 20:19 UTC
    Give Logfile::Tail a spin:
    #!/usr/bin/perl use strict; use warnings; use Logfile::Tail (); use Data::Dumper::Concise; my $file = new Logfile::Tail('/var/log/messages', {autocommit => 1 }, ); my $line = $file->getline(); $file->commit(); print Dumper( $line );
Re: Reading constantly generated log files: Suggestions invited
by pvaldes (Chaplain) on Nov 08, 2013 at 20:53 UTC

    You obtain all the names fitting in a date in the dir, you sort the files, you pick the first and break the loop. This is more or less the idea

    I had the same problem in one of my small projects, solved using a system line like:

    system('for i in `ls -r $(date +%F_%H\:)*.log | head -1`; do emacs $i; done');

    Always opening the most recent file as long as the timestamp in its filename is right

    (Entire script, just in case you want to read it, here)

Re: Reading constantly generated log files: Suggestions invited
by aitap (Curate) on Nov 09, 2013 at 19:05 UTC
Re: Reading constantly generated log files: Suggestions invited
by Random_Walk (Prior) on Nov 09, 2013 at 15:06 UTC

    I had to do this once, with the added fun of arbitrarily complex processing of the log lines. I used POE::Wheel::FollowTail http://search.cpan.org/dist/POE/lib/POE/Wheel/FollowTail.pm which worked a treat. Probably overkill, but it opens a whole world of event based log processing fun to you. I handled some busy weblogs and as well as looking for errors, produced a pile of stats every minute too

    Cheers,
    R.

    Pereant, qui ante nos nostra dixerunt!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1061736]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (3)
As of 2024-04-24 21:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found