|No such thing as a small change|
Reducing memory usage while matching log entriesby matt.tovey (Beadle)
|on Feb 01, 2006 at 14:09 UTC ( #527058=perlquestion: print w/replies, xml )||Need Help??|
matt.tovey has asked for the wisdom of the Perl Monks concerning the following question:
An application-programmer colleague of mine, noting that I program in Perl, asked me for a script to help him analyse log files. How could I refuse?
Problem is, my solution needs the whole logfile in memory, and these things can be quite large (100MB), so I'm wondering if there's a better way. No, strike that, I'm wondering what the better way is. :)
The logfile consists of the locking and unlocking of various mutex's, where my colleague wants to disregard (i.e. remove) all references to locks which are later unlocked. I'm doing this so:
Nearly all of the locks that are set will be cleared close to the time that they are set, so the memory-use could be significantly reduced by a solution that frees memory no longer required. I was (very optimistically!) hoping that setting array entries to '' would free the memory, but a bit of profiling shows me that it's not so.
So, what would be a better solution? Here are some ideas that I had:
- splice lines no longer required out of @lines.
- grep backwards through @lines when I find an 'unlocked' message.
DEBUG : MUTEX : : Mutex(0x30080002 + 0)::locking DEBUG : MUTEX : : Mutex(0x30080002 + 2)::locking DEBUG : MUTEX : : Mutex(0x30080002 + 2)::unlocked DEBUG : MUTEX : : Mutex(0x30080002 + 4)::locking DEBUG : MUTEX : : Mutex(0x30080002 + 2)::locking DEBUG : MUTEX : : Mutex(0x30080002 + 2)::unlocked DEBUG : MUTEX : : Mutex(0x30080002 + 4)::unlockedIn this case, all but the first line should be removed.
The contributions from hv and meetraz are much more efficient, and run much faster too. Thanks! However, I'll have to ask my colleague if he actually needs to see the lines which weren't removed (which I simplified somewhat in the sample data above - sorry!), or if such a report on the locks will suffice.
Update 3: :(
Back to Seekers of Perl Wisdom