Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Work practices: log books, notes files...

by doom (Deacon)
on Apr 16, 2008 at 21:34 UTC ( [id://680926] : perlmeditation . print w/replies, xml ) Need Help??

talexb wrote (in Interview Counterattacks):

I started keeping a log book a few years back -- it was something my colleague Mike Stok did, and it's fantastic. Now when someone asks me something, I flip back in the book and read out the command I typed. That's way better than "Oh, I think I typed in ..". I can also pin-point, to day anyway, when I actually did something a few weeks or months back.

Paper log books have their uses, but myself I've gotten in the habit of using text files full of notes (and I adopted the convention of naming them all "notes-*", very useful when I want to search for them). As I'm working on a project, I add things to the notes-* file for the project (error messages, thoughts, commands I tried, snippets of code, solutions that worked). It helps somewhat that I tend to live inside of emacs: I may have a shell window and a notes-* file buffer open in the same emacs frame, so I can switch back and forth between the two of them. The notes-* file helps me focus even if I'm interrupted/bored/etc and if someone says "Hey, you know how to do this, you did this last month, right?", I much prefer being able to say "Let me check my notes" rather than "Last month! You expect me to remember what I did last month?!"

Further -- since I work in emacs -- I can save the shell buffer to a file (usually called "shell-*") and I always have a record of precisely what I did, even if I didn't think to add it to the notes file. There was one occasion where my supervisor asked me if I might have forgotten to do a COMMIT the day before, and I happened to have the SQL/PLUS session saved to a file, so I could prove that I hadn't forgotten.

The drawback to this, however, is that if someone accuses me of causing a problem by a stupid mistake, it's possible for me to verify that yes, indeed, I am the one who made that stupid mistake.

More about notes-* files practices, if any one cares: I type rows of equal signs (e.g. "===") to visually divide between topics; I often label things with uppercase tags (e.g. "TODO", "SUMMARY", etc) that are easy to find with case-sensitive searches later; and I've got a datestamp command bound to a few keystrokes, which I use nearly every time I start working...

Replies are listed 'Best First'.
Re: Work practices: log books, notes files...
by almut (Canon) on Apr 17, 2008 at 00:18 UTC

    Nothing spectacular... but I've gotten into the habit of storing away my current shells' command histories whenever I feel I've done something non-trivial (debugging/compile session, etc.). For that I have a little script which essentially does history >~/memos/YYYY-MM-DD_hh-mm-ss.hist (YYYY-... being the current timestamp). Occasionally, I clean them up a little, or add a few short notes, but more often I don't.

    I can then easily grep through those files whenever I can no longer remember what I did when and how...  Also, I can cut-n-paste stuff (can you remember all the hostnames and portnumbers you typed two months ago when you had set up that three-hop SSH tunnel to some client? — well, I can't). The possibility of chronological access (kind of like a 'diary') has sometimes turned out useful too, to reconstruct things — when someone asked or came up with strange claims...

    All in all, this has proven very useful, considering that it takes almost no extra effort once you've made it a habit... (which is important for the generally lazy chick that I am :)

      My zshrc has ...

      date=$( date +'%Y%m%d' ) # HOSTNAME is assigned, well, host name elsewhere. HISTFILE="${HOME}/log/sh/zhist.${date}.${HOSTNAME}" SAVEHIST=9000 HISTSIZE=9500

      I've gotten into the habit of storing away my current shells' command histories whenever I feel I've done something non-trivial (debugging/compile session, etc.).

      You could script that and run it from a nightly cron job, so you'd have a library of shell command histories to refer back to, irrespective of whether you thought you were doing something important at the time...

        I'm sure I could, but I prefer to have it filtered through some conscious, deliberate selection process — not everything I type in any shell I open is worth being preserved... and storing everything would likely only reduce the signal-to-noise ratio. Also, the semi-automatic approach encourages me to add notes or keywords, which tend to improve the overall usefulness of the history files (you know, when you're lazy, you have to devise ways to trick yourself into doing things you otherwise wouldn't do...:)  And honestly, spending one or two seconds when you've finished something (or before you close a shell) to decide whether it's worth being preserved, is not too much of an issue for me. But, as always, YMMV.

        Actually, the tools I use to 'manage' my history files are slightly more flexible than what I mentioned above (I can specify how many most recent entries to save, whether to create a new file or append it to some other, whether to open it up in my editor, or filter out common trivialities, etc.), but I didn't want to bore you with the details...

        BTW, other than that, I'm pretty old-fashioned in that I very much like paper and pencil to sketch design diagrams or write down temporary todo lists, etc. For one, it just feels better to be able to physically cross out completed items, rather than simply deleting the respective entries in an electronic version (yes, you guessed it, I'm a little weird at times :)

Re: Work practices: log books, notes files...
by wade (Pilgrim) on Apr 16, 2008 at 22:19 UTC

    I recently started using a personal version of the MediaWiki software for my notes. It's free, easy to use, and available from

    I keep a blog in the wiki where my daily notes go but that blog refers to 'project' pages, 'tool' pages, and 'process' pages. I have templates for regression tests, the results of which go into my wiki. I also have a glossary (which was great when I started a new job, recently) and a list of people (with links to the projects on which they're working). Even better: if I don't remember where I've put some notes, I can search. I've come to like this method of note taking so much that I have started taking my home notes on a thumb drive running this same software.

Re: Work practices: log books, notes files...
by Limbic~Region (Chancellor) on Apr 17, 2008 at 01:45 UTC
    Not me.

    I had a coworker and friend over a decade ago who swore by them. On the cover, he always wrote "Shadow" and any time any one asked him a question, he always said "The Shadow knows" and would thumb through until he found it. This guy had journals going back over 20 years. He was an incredibly smart guy.

    It never rubbed off. I rely on my memory almost exclusively.

    I almost never walk into a meeting with a paper and writing utensil. If I do, it is likely for show. When I do spend the time writing down instructions, they are extremely detailed - but it is done for other people. I am going to be 2 ** 5 this November and people keep telling me that your memory starts failing you as you get older. Funny thing is, people have been telling me that all my life.

    I remember being about 10 years old and quizzing my step-mother if she knew the order of the planets forwards and backwards. Her reply was - I used. Let's see if you still know them so well in twenty years. I was sure to remind her last year that I still knew them forwards and backwards despite pluto no longer being considered a planet. I am often the "go to" source when someone needs to recall something. I have a good memory and rely on it.

    On the other hand, my memory seems to depend on adequate sleep. That's something I haven't really been able to get reliably in about the last 6 months. Recall time is slower and inaccurate. Sometimes it is worse to to misremember then it is to not remember at all. When I absolutely do need to write something down - it is in email. Email is the perfect medium for me.

    Email for note taking allows you to "remember" using any number of hooks (Who, What, When, Where & Why). It also is useful if you want to break up a body of work in time intervals for the purposes of writing status reports :-)

    Cheers - L~R

      Ten years ago - even eight years ago - I would have responded exactly as you did; I was just as proud of my nearly-perfect memory as you appear to be. I could reel off long passages from books that I'd read years before (ones I liked, anyway) and I never needed an appointment book. Then, one fine day - more like over a period of a month, actually - my memory crashed and burned and died. I can't even begin to tell you what a painful and terrifying experience that is, every single waking minute for years afterwards, when that happens - and when you make your living by using your mind.

      These days, I have a memory that works pretty well in some aspects, is rather poor in others, and is absolutely horrible in a a few. I still feel like screaming, regularly, when I reach for something I "know" I should have and hit a blank wall.

      I don't know if this is a common experience for people who have good memories, but for your sake, I hope that my experience was an anomaly.

      Update: In case you're curious, I'm 46, and lost my mind (?) at ~39. No serious drug or alcohol usage of any kind, no brain injuries, no physical reason for it at all. Had a relationship die a horrendous death that year, but that was about all.

      Human history becomes more and more a race between education and catastrophe. -- HG Wells

        Very interesting post.

        Though I wonder if it happened quicker than usual for you - and that due, in part, to a dire year at about that time of life. I'm 36, and am slowly finding my memory is not managing everything. I'm sure it's partly because I've more projects/things on the go than I used to, but...

        Now what was I going to say? Heck, forgotten, never mind :)

        I'm sure another lesson is that life quality (health, sleep, happiness) affects our brain performance, big-time. Also, if I'm not interested, then my IQ does a bit of a downward slide. Reversible, I hope.


        When did it start to happen? Well, I can't quite recall ... :-D ... All kidding aside, it does happen.

      I'm envious :).

      My memory is terrible. All tasks in school that relied on rote memorization were very taxing for me, and I can honestly say I remember very little of what has happened in the last week.

      At all times, I keep a small hardcover notebook with me, and use it to record everything. Thoughts I have, tasks to perform, all sorts of reminders to myself. If I didn't have it, nothing would ever get done, just thoughts drifting into the mist :).

      The next exponent is uncomfortably close now. It doesn't get any easier.

      I keep a paper log, in a loose-leaf binder that sometimes contains printed pages. I don't always keep it at the workplace, preferring instead to read it in the morning and to update the latest notebook in the evening. When I write in it, I use a number-two pencil and a big fat artist's eraser. It slows me down, and I do that on-purpose.

      You forget a lot of things. We all do. That's why documentation is so important. I was recently asked to unearth some code that I'd written in-preparation several months ago and to quickly put that code into service. Well, the first thing I had to do was (literally...) to find it. Then, I had to manage to understand it in only a couple of days. It was tough, but I did it.

      • “Find” it?
      • “Understand” it?
      • “In only a couple of days?”
      • It was “tough?” (Like, dude, didn't you write it?!)

      Yes, of course I did write it, but when confronting that work today ... it was the work of a complete stranger. I was at-first no more equipped to deal with that code than anyone would have been who was seeing it for the first time, as I for all intents and purposes was.

      Fortunately, “the person who wrote it” left lots of notes. In the version-control system there was a complete design-document, which I re-read. And there was the logbook. And finally, lots of comments in the code. These sources pointed me to other related systems that I had not at first (re-)discovered. Most assuredly I would have been floundering around for a week or more just getting my feet on the ground, and there was not “a week or more” to be had.

      I don't think that any of this has anything to do with that “next exponent.” Rather, I think that a lot of us have very sloppy, very slap-dash work practices (and plenty of excuses and plenty of books to say that this sort of thing is par-for-the-course, or necessary due to the nature of the work, or even brilliant because we're all such geniuses...). And, yeah, I say “we.” A self-inclusive pronoun.

      Engineers in other professions are taught and required to keep lab-notebooks. But we're not. Companies plying other professions maintain project-plans that are thousands of items long:   we attempt to substitute obstinacy.

      We should know better. They do...

      I am going to be 2 ** 5 this November and people keep telling me that your memory starts failing you as you get older.

      Patience, Grasshopper. 32 is still quite young. By 0x32 you may start to notice some changes. Mostly to do with things recent. I still know the Main Sequence of stars (OBAFGKMRNS), resistor codes (Black, Blue, Red, Orange, Yellow, Green, Blue, Violet, Gray, White), planets (ha!), but don't ask me exactly how I set up the test learning management system on Sharepoint last week.

Re: Work practices: log books, notes files...
by jplindstrom (Monsignor) on Apr 17, 2008 at 00:14 UTC
    Since you use Emacs, you should definitely check out org-mode.

    Don't start with all the features at once, just getting going with the outliner and table mode makes a great deal of difference to how you manage your notes.


      Hm, and it looks like org-mode is part of the Gnu emacs distribution these days. I'll have to look into it... I'm not really interested in changing my system exactly (completely free-form text files plus find/greps are pretty killer, if you ask me), but it could be I could use a better way of managing TODO lists.

      And having a spreadsheet available using lisp formulas could be cool.

      Ooh---I didn't know about that. I'm just getting started with Emacs, but I'm liking it a lot at the moment. As for organisation/documentation: I tend to document what I'm doing as I go along in files called 'notes', though my system is rather patchy to say the least at the moment. I also have a normal 'lab book' which I keep as a kind of scribbling pad for ideas, writing down questions (and their answers) and generally getting more of an overview of things. Also I use it for preliminary calculations worked out by hand, and diagrams etc. There's something very satisfying I find about filling up a book with scribbles of your work, and very handy when you forget what you've been doing.
      Those are my principles. If you don't like them I have others.
      -- Groucho Marx
Re: Work practices: log books, notes files...
by talexb (Chancellor) on Apr 17, 2008 at 03:01 UTC

    Interesting .. however, my notes can go with me into a meeting and I can refer back to them at any time. I can discuss them with someone else in the kitchen. I can write stuff down while sitting at someone else's pod, and someone else can draw a picture on a page. For instance, I like block diagrams -- those are way too time-consuming to do in a text file. And if I need a common line history, I print it out and staple it to the appropriate page.

    But -- to each his/her own. Use whatever works for you, I guess.

    Alex / talexb / Toronto

    "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

      For instance, I like block diagrams -- those are way too time-consuming to do in a text file.

      Well, now that you mention it, I'm pretty good with emacs picture-mode too, but your point is taken that paper has it's own advantages. I'm not saying it doesn't.

      (I am not, for example, a PDA kind of guy: I've always got a moleskine in one side-pocket, and a slingshot in the other).

Re: Work practices: log books, notes files...
by TimButterfield (Monk) on Apr 17, 2008 at 13:05 UTC

    I have long done something similar, though not quite to the same extent. I don't save every command, just ones I feel relevant or wish to repeat combinations of. In addition to dated text files such as yyyymmdd.sql, I have also used a single file wikis like GTD Tiddly Wiki Plus to make it easier to organize notes by project. Here is the 43 folders description page and a direct link. It provides an easy way to keep things organized and is easy to carry around on a USB memory stick.

      blosxom anyone?

      It's pure txt files and Perl!
Re: Work practices: log books, notes files...
by bwelch (Curate) on Apr 18, 2008 at 13:31 UTC
    I've been using three-ring binders to take work notes for a few years and eventually evolved into keeping a daily work log. Writing paper notes was too slow for me, though, so I started using a text editor and found that was much better.

    The only issue was finding things. Searching thousands of lines of notes for a partially common idea or term didn't work as well. That led me to look into using something hierarchical where I can maintain a tree of ideas.

    Looking around, I like Treepad Lite (freeware). It's working well for organizing notes, but at the same time it's more work to maintain. I might end up going back to a full log and searching it, or I might end up copying the work log notes into Treepad every week or so to have things organized.