|
jeepj has asked for the wisdom of the Perl Monks concerning the following question:
Hello monks, I am yet again in need for some guidance.
Very often, from my pov, the question of log format is only put when the log is there and we have to parse it. I have the opportunity to decide the log format for new Perl scripts, and I seek you advice to choose the best solution
I am logging outputs from a mainframe, and for each reference, I have to log the main screen of the record, and depending on the need, I have to issue several commands on this record and log them. For the moment, I am only logging outputs in a flat text file, each record being separate by a line of '-', and inside each section, each entry is starting with a line like '>MD' where '>' is the prompt and 'MD' a command (example).
I am now looking for a smarter way to log the output, but I guess that it could be also useful to log the actions taken on the record, and various information technical and functional extrapolated by the script. Instead of using a flat file, I am interested by the XML format. What do you think ? is there a better format ? also, what would be for you the ideal format to mix outputs and other information in a single log file?
As a minor question, what is for you the best module to write and read simple XML files used for logging activity?
|
Re: Efficient log format
by dragonchild (Archbishop) on Mar 18, 2008 at 13:52 UTC
|
The trick, I think, is to not try and make a format that does both efficient for writing and efficient for reading. Instead, create a format that's easy to write to. Then, when you need to read from it, you do so. In most cases, logs are always written to, but rarely read from.
AND ... if you correctly encapsulate the writing of the logfile from the rest of the code, you can change the format later, as needed.
My criteria for good software:
- Does it work?
- Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
| [reply] |
|
|
I'm afraid I can't agree with you. Firstly because in our context, I'm speaking about logs, but these are not like servers' logs, this are results from analysis/correction scripts, and results are important, verified, and re-used in some cases to take further actions. But I was propably not clear enough on this point (and the word 'log' is perhaps not the better one in this case, but english is not my mother tongue, sorry ;) )
Secondly and more generally, I think that putting in place a good and well formatted logging does not require a lot of efforts if done at an early stage. This ease the addition afterwards of new information and the retrieve of this information by other processes.
I however fully agree that this must be encapsulated in a separated library, to ease the log of information in the scripts. But I'm not sure that changing the format later is easy, as you have to translate all past logs to the new format
| [reply] |
|
|
Yeah, that's not a log. That's a "result" or "product". You want to think those through. Confusion sucks. :-)
My criteria for good software:
- Does it work?
- Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
| [reply] |
Re: Efficient log format
by herveus (Prior) on Mar 18, 2008 at 13:34 UTC
|
Howdy!
How about using SQLite to hold the logs. One table with a suitable set of columns
to hold the parts of a log record might be all you need.
| [reply] |
|
|
Thanks for the suggestion, it's a good one, but for a lot of reasons (software complexity, maintenance, company policy regarding third party software...) I want to keep the whole process based on Perl scripts and the PCOMM software from IBM.
| [reply] |
|
|
Howdy!
OK. I don't know PCOMM at all, but I was presuming that one would use DBI to interact with
SQLite to post entries to the log. One could develop log reading tools in Perl, or use the
command-line interface that comes with SQLite to examine the logs.
What I meant by my suggestion was to use a real database instead of a flat file for the
storage mechanism.
| [reply] |
Re: Efficient log format
by Jenda (Abbot) on Mar 19, 2008 at 23:04 UTC
|
If it wasn't for the great and inteligent restriction that the whole XML has to be enclosed in one tag otherwise it's not legal XML, I'd say go ahead and use XML. I mean you might append the closing tag every time you write something and trim a few characters from the file whenever you need to append something, but that's tedious and error prone. So I guess ... use something that's not XML according to the purists and either append the closing tag (that's a term you are not supposed to use either) before you attempt to parse the file or split it into records before parsing the individual ones as if each was a separate XML.
I'd most likely use XML::Rules for reading and a combination of plain old print "<tag>$escaped_value</tag>"; and XML::Rules for output.
| [reply]
[d/l] |
Re: Efficient log format
by jeepj (Scribe) on Mar 20, 2008 at 10:20 UTC
|
First, thanks to herveus, dragonchild and Jenda for your comments, they allowed me to progress on my problem
Regarding XML, I have made a simple test, by using simple print, and XML is finally not the solution for me: some of my outputs contains '&' characters, and I can't escape them as I want to keep my outputs untouched.
I have also looked at structured text solutions, like wiki format or others, and the conclusion is that they don't fit my needs. In fact, I only need a way to separate outputs taken from my 3270 sessions, with only the command issued to get them. As 3270 screens are limited to 80 characters, I will use line separators of 81 characters, containing the information required to understand what is logged after. This is pretty simple, but will fit my needs. And as mentioned by dragonchild, putting the code to write and read the outputs will allow me to enforce the format, and make it extensible.
| [reply] |
|
|
