Re: Parsing a specific section of a log file

If at say 10:25, the situation at 10:20 is known, it obviously can't change five minutes later. Also if the log file is large, the File::Tail will perform better than the implicit costs of closing, reopening and reading backwards every time anyway. The advantage of that one over the POE solution is that you can sleep for 300 seconds in between to minimise consumption of system resources, whereas POE will keep itself somewhat busier, though not as busy as reading backwards every time.

But it often helps in busier systems to divide very regular processing into small slots by design, so that the next daemon to be implemented has a slot available whose consumption by other processes can reasonably be predicted - that can't be done for the POE solution.

-M

Free your mind

Comment on Re: Parsing a specific section of a log file

Replies are listed 'Best First'.
Re^2: Parsing a specific section of a log file by eriam (Beadle) on Mar 01, 2007 at 10:50 UTC
I don't see why you can't do that with POE. If you have other processing to do once in a while just add it as an event in your main POE processing daemon (which is fo example tailling a log file up to now) and fire it up as often as you wish. Then it will hopefully take full advantage of POE's cooperative behaviour. Honestly implementing POE is an effort (small) but as long as you build the daemon from scratch I think it's a good choice (and you don't need cron jobs..). Anyway there is more then one way to tail a log file :)	[reply]
Re^3: Parsing a specific section of a log file by Moron (Curate) on Mar 01, 2007 at 12:33 UTC
Event-driven processing is distinct from scheduled processing and consumption distribution can only be managed from scheduled processing. In addition, not every process on a system is in general written in Perl. Nor does one generally have the luxury of rewriting a system just because, say, one believes event-driven processing to be aesthetically better - that is at least how managers tend to view it. In my own situation, the only way I can achieve a totally event-driven version of the kind of system I work on is to pay for the development myself. -M Free your mind	[reply]
Re^2: Parsing a specific section of a log file by Random_Walk (Prior) on Mar 01, 2007 at 16:51 UTC
I use POE::Wheel::FollowTail to parse, analyse and forward events concurrently from two very busy logfiles (each about 100M a day, average 10 significant lines per second, picked out of three or four times that many, peaks way above that). I have POE look at the file every five seconds and the CPU hit is negligable compared to other processes on the box. I guess it queues itself an alarm between checking the file as this particular POE is doing little else (throws an I'm alive event every minute just in case). I like the fact the FollowTail handle the log file being rolled over, tells me about errors without choking and generaly just does the right thing. It saved a lot of thinking about all the corner cases (in my case these files are r/o NFS mounts from elswhere and occasionly mis-behave) and let me concentrate on the interesting bit. Cheers, R. Pereant, qui ante nos nostra dixerunt!	[reply]


Perl Monk, Perl Meditation
	PerlMonks