Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Tips for managing Perl projects?

by radiantmatrix (Parson)
on Dec 15, 2004 at 18:27 UTC ( [id://415138]=perlquestion: print w/replies, xml ) Need Help??

radiantmatrix has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, wise ones.

I find myself in a unique position; my current client has asked me to develop guidelines for all future project management involving Perl (since I am, frighteningly enough, the local Perl expert). This is not a development shop, but tools for administration and automation are regularly developed. The organization is standardizing on Perl.

Now, of course I know the obvious:

  • Require unit tests
  • Store all documentation and code in the RCS
  • Backup, backup, backup

I read Random thoughts on programming (thanks, tilly), but what I would like to know is what tips the Monks wiser than I may have about project management and 'best practices' for small to medium Perl projects. Are there neat Perl tricks anyone uses to help with such things? What should I require (beyond use strict; use warnings;) for code spec?

My management style tends toward making as little restriction as possible, so I'm not about to provide a laundry list of requirements. But, I would most sincerly appreciate a bit of enlightenment, if only in the form of what you have experienced and the positive and negative aspects of them.

Perl was chosen as the standard based on the combined recommendation of the former team lead and me (even though I was contracted only to admin a cluster); now that he left for greener pastures, it is left in my lap to write all the standards and best-practice documents.

I thank you all in advance for your wisdom.

radiantmatrix
require General::Disclaimer;
s//2fde04abe76c036c9074586c1/; while(m/(.)/g){print substr(' ,JPacehklnorstu',hex($1),1)}

Replies are listed 'Best First'.
Re: Tips for managing Perl projects?
by dragonchild (Archbishop) on Dec 15, 2004 at 19:17 UTC
    Note: This list assumes peer-reviewed design, if not peer-reviewed code.
    1. Before writing new code, demonstrate that what is needed cannot be found on CPAN.
    2. Before writing new code, demonstrate that what is needed cannot be found in-house.
    3. Attempt to write large modules and small scripts that use aforementioned large modules.
    4. If you want to disable strict and/or warnings, you better have a damn good reason.
    5. It's better for the code to run 10% slower than for the coder to take twice as long. Code reuse is good for a reason.
    6. Standardize to a common set of idioms / modules / processes for the following items:
      • Configuration files
      • Database connectivity
      • Logging
      • Exception handling
      • Documentation (not everyone should use POD, believe it or not)
      • Source code rollouts (how should you install something?)
    7. Taking an extra hour before touching a keyboard will save you a hundred hours in the next 6 months. To truly be Lazy takes a lot of preparation.

    And, my favorite recommendation is Keep It Stupid-Simple ... so simple even stupid people can understand. Cause, frankly, the guy after you will most likely be stupider than you are.

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

      As a "QA" tester (I hate that name, but it points towards what I actually do), I'll emphasize:

      Logging
      Exception Handling (especially for I/O)

      If you have to, you can do the rest after the fact, but if you lack either of these, fixing things becomes radically more difficult.
Re: Tips for managing Perl projects?
by Elgon (Curate) on Dec 15, 2004 at 19:19 UTC
    Hi Radiantmatrix,

    Unfortunately, I am very bondage-and-discipline when it comes to project management, partially because I work for a large consulting firm and am used to working on very large projects with many coders of widely varying ability, but also because I have a very logical mindset about this sort of thing. If any of the advice which follows clashes with your style, please feel free to use it or disregard it completely as you deem appropriate. This ties in with my first principle;

    1 - Be consistent where possible, but always manage to the appropriate level.

    For a big project which is to be handed over to an external client, it makes sense to go for the full-on document-everything-and-get-signoff approach. On the other hand, if you've only got two developers and one deisgner working for you on a small internal gig, then a simple one-page spec may well be the ideal way of communication, with much of the more detailed documentation being in code comments.

    The logical end point of this rule is that you may ignore any other rule or create any new one, provided that it is the right thing to do.

    2 - Keep all of your emails and all of your documentation in a central repository for later reference.

    This comes under the heading of covering your arse, which, however unfortunately, is a very necessary evil of being a project manager. The ability, when challeneged, to be able to pull out an email which covers the main point of contention is invaluable. Ideally all of these would also be kept under RCS.

    Also remember that documentation isn't for you or your team members. It is for the people who have to use and support the tools that you build. Even if this is your team, just remeber that in five years time, when your widget is still being used, it is highly likely that you and the rest of your team will no longer be there.

    3 - Set a few simple design standards.

    These will vary, but given that you're talking about Perl specifically; An example principle would be something like "Any function which is used more than twice in a script should be declared as a function" or "Any function which is used more than twice in different scripts should be included in a library module".

    The most important design principles of them all are - "Use some common sense" and "Keep it simple, stupid."

    4 - Set a few coding standards.

    As above with design standards, coding standards will also vary according to your preference. For Perl, I would suggest that use strict; and -w should always be the case, except where there is a good reason not to. Make sure that your coders don't break APIs when they "fix" things.

    5 - Revision Control

    Use it and make sure that version control discipline is enforced. There is nothing more irritating than finding three different versions of the same code, each with slightly different filenames in the same RCS. Forking hell!

    6 - Deadlines and Caveats

    When asked to give work estimates and set deadlines, give a possible overrun and state all boundary conditions. Sadly, this also often comes under the covering your arse heading, and people want what they want yesterday and for free. Unless you wish to blow your time and fiscal budgets, make sure that the people for whom you are setting the deadlines understand a) that there is an error factor involved, and b) that if they change the spec then the deadline is also going to change.

    Task = Time * Resource

    This equation is not generally mutable for programmers and managers of given skill, so you can't just double the size of the task without increasing the Time taken and/or the Resources available.

    7 - Testing

    For every hour of time spent on actually writing code, you're likely to spend a day on writing or executing tests. This is a pain and only the truly anally-retentive enjoy testing, so try to automate as much of the process as possible. A couple of point on testing generally...

    • Test negatively as well as positively
    • Unit test each component
    • Test all of the components in an assembled state, but don't just rerun all of your unit tests again
    • Use realistic test data
    • Use realistic volumes of test data

    Just a few suggestions, and as I say at the beginning, please feel free to implement or discard as you prefer. Good luck.

    Elgon

    It is better either to be silent, or to say things of more value than silence. Sooner throw a pearl at hazard than an idle or useless word; and do not say a little in many words, but a great deal in a few.

    Pythagoras (582 BC - 507 BC)

Re: Tips for managing Perl projects?
by BrowserUk (Patriarch) on Dec 15, 2004 at 22:12 UTC

    One word phrase:

    Code reviews

    It's not a Perl specific thing, but is applicable to every code project outside of the one-man development effort.

    Code reviews should be quick and painless affairs. More than two reviewers of one peice of code will often lead to arguments between the reviewers; and review time increases with the square of the number of reviewers; so one-on-one is best.

    The reviewer, reviews the code on their own and makes notes about anything

    • They don't understand.
    • They think is an error.
    • That they would do/have done differently.

    The reviewer/reviewee get together. The reviewer steps through each of his/her notes explaining: What they didn't understand; Or why they think there is an error; Or what they would have done differently, and why.

    The reviewee listens, and takes their own notes. At the end, the reviewee gets the opportunity to ask for clarifications, if they are needed.

    The review is over.

    If done weekly, the process should take no more than 1/2 hour of meeting time, and a coffee break or two for the reviewer to review latest work.

    There is

    • No scoring.
    • No rating.
    • No "you must fix/change/redo".
    • No explainations cross-examinations* of why the reviewer would do it different.
    • No explainations cross-examinations* of why the reviewee did it the way they did.
    • No arguments.
    • No arguments.
    • No recriminations.

    *Updated to correct the ambiguity as pointed out below by itub. I'm not sure that this change reflects what I am trying to say all that well either, but I can't think of a better phrasing. It's about stating reasons without drawing conclusions or offering supporting argument. The desire is to make statements of what and why as simply as possible, and then allow the other party to draw their own conclusions. Of course, the discussion will likely resume later, and become detailed around the water cooler, but the hope is that outside of the formal review procedure, any argument that ensues does not taint the review procedure itself.

    The code reviews should be scheduled, regularly and not routinely deferable. People should not get into the habit of "cleaning up their code for reviews". Whatever they have at the allotted time is what gets reviewed. This encourages the habit of keeping code fairly clean at all times.

    Everything you write should be open to review. Ideally, even the standalone quick tests of ideas, as a lot can be learnt from the way people try things out too. This is harder with Perl where so much can be tried out with one-liners and/or perl -de1 and the like.

    The review process is to give the reviewee an insight into how others read his code, and to benefit from alternative ways of tackling problems. It is up to him to institute any changes in the light of what he learns.

    Having your best coder review the weakest/least expereinced/newest team members code--and vice versa--is a pretty good way to ensure that your that the weakest get stronger, and the strongest don't get carried away.

    Re-reviews of code only happen when a developer says his code is unit tested and ready for integration.


    Examine what is said, not who speaks.        The end of an era!
    "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
    "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
    "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
      I'm confused. First you say that the reviewer should explain what he would have done differently and why, and later you say that he shouldn't.

        Your right. That was ambiguously and contradictively phrased. I've updated in an attempt to clarify, though I'm not sure I've suceeded?


        Examine what is said, not who speaks.        The end of an era!
        "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
        "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
        "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
Re: Tips for managing Perl projects?
by freddo411 (Chaplain) on Dec 15, 2004 at 18:36 UTC
    Speaking of tests, wouldn't it be a good idea if the rules you came up with were instantiated as a set of tests that would run against the scripts in question?

    Test/Rules that might make sense:
    * Require use strict and warnings
    * Require pod
    * Perl tidy??

    Sometimes it is better to NOT have too many rules, but some "best practices" can be encouraged by providing easy, documented access to helpful tools. For example, if you have a decent logging module that your firm used by convention that would be good.

    -------------------------------------
    Nothing is too wonderful to be true
    -- Michael Faraday

Re: Tips for managing Perl projects?
by Ovid (Cardinal) on Dec 15, 2004 at 22:26 UTC

    Why do large projects have such an abysmal failure rate? Large companies that can afford to throw a half million or more at a project don't do so on a whim. They spec out the project and determine if it's feasible. And despite this, it still fails. Why?

    It fails due to information costs. Those costs are frequently not considered when the project starts and the larger the project is, the greater the informatin costs. For example, Alice is the boss of Bill and Charlie. Alice happens to tell Bill something that Charlie should know and expects him to tell Charlie. Bill tells Charlie Bill's understanding of what Alice said and Charlie writes bad code because his specs are wrong. However, he doesn't know to ask Alice what the correct specs should be because he had no reason to think he was heading down the wrong path.

    Obviously, if you have one manager and only two programmers, this is less likely to happen. However, as a project grows, these sorts of miscommunications continue to occur and the poor Charlies realize something is wrong so they keep trying to double-check their information and this raises overall costs (which is why I refer to "information costs", sometimes known as "search costs").

    Part of the reason XP is so successful is that it defines a lightweight system where everyone has a clearcut role and you know exactly where you should turn for any information you need. Everything in successful project management is about lowering these search costs. Any time someone has a question, they can immediately receive timely, accurate and complete information. If you can pull that off, you'll probably be a successful project manager. Mind you, this isn't saying you should go the XP route, but if you do, it's frequently a quick route to success.

    Cheers,
    Ovid

    New address of my CGI Course.

      I agree with your observations but...

      As far as I understand it XP is mostly about eliminating project management by having the client sit down next to the programmer and telling him directly what he's expecting from the program.

      I'm not saying that's a bad thing; most programmers could certainly do with more direct communication with the client. I do think it doesn't solve the problems in a project that's already running - the client simply never has the time to do this. Last project I did that went horribly wrong, we had more than 10 programmers working at the project 8 hours of every working day - we were lucky if the client could spare one person 4 hours per week to help us out with the specs. Ofcourse almost all of that time was spend by management promising the client golden castles in the sky instead. This particular project went over the deadline by more than 20 man-months (original estimate: slightly over two man-months - yes, the estimates were that bad)

      And ofcourse, the client expects the project managers & programmers to be clairvoyant anyway :-).

      I think most of the problems in the project described above were caused by just plain awful specs and cost-estimates coupled with horrible management from all sides.

      I think the most important lesson I learned from that project was that you simply cannot spec & build anything without really understand what the client wants - not what they tell you they want: what they really want - what their goal is, how much they want to spend on it, and what their priorities are.

      Let's back up a bit from there: I think I've spotted a recurring theme from all the projects I've done that went bad (as in: over budget and/or made me feel terrible and stressed out)

      The most important thing you need from a client is a list of functionalities ordered by priority. Get this list before you start designing. It will make the client think about what it is they want, and you can (hopefully) see from that list what it is they actually think is important about the product. At least 50% of the time, you'll be really surprised (and then you can probably talk him out of paying for weeks of development on that really annoying piece of functionality that the client actually doesn't give a damn about, but was just put in the specs to stop somebody nagging in a meeting).

      I quit that job by the way - I feel much better now that I'm self-employed :-)

        As far as I understand it XP is mostly about eliminating project management by having the client sit down next to the programmer and telling him directly what he's expecting from the program.

        This is not eliminating project management. It's about improving information flow. There's still a project and the work is still being managed. It's just not being managed with a bunch of charts, out of date documents or unwanted features that are hidden away until they are revealed a few months past the deadline.

        Just because it's not being done the old way doesn't mean it's not being done (or that the old way is necessarily bad, for that matter.)

        Cheers,
        Ovid

        New address of my CGI Course.

        the client simply never has the time to do this.

        That's easy. The client really doesn't want working software.

Re: Tips for managing Perl projects?
by b10m (Vicar) on Dec 15, 2004 at 18:43 UTC

    You could require the use of perldoc perlstyle (or make variations on that). Also require proper documentation. It's rather interesting to know what some code is supposed to do, rather than reading the obvious (which can be found in the source easilly).

    --
    b10m

    All code is usually tested, but rarely trusted.
Re: Tips for managing Perl projects?
by sauoq (Abbot) on Dec 15, 2004 at 21:48 UTC

    Get perltidy. Use it.

    -sauoq
    "My two cents aren't worth a dime.";
    

      Perltidy is great for managing the appearance of one's code, but it doesn't begin to address the management of an entire project.

      radiantmatrix
      require General::Disclaimer;
      s//2fde04abe76c036c9074586c1/; while(m/(.)/g){print substr(' ,JPacehklnorstu',hex($1),1)}

        Then it's clear you've never watched endless discussions of where to put the damn curly braces during peer review. I would repeat the admonition, get perltidy, use it.

        --hsm

        "Never try to teach a pig to sing...it wastes your time and it annoys the pig."
        it doesn't begin to address the management of an entire project.

        Uhm... of course not. I think most people might realize that I wasn't trying to address the management of an entire project in my four word reply. I was providing a single "tip"... nothing more.

        -sauoq
        "My two cents aren't worth a dime.";
        
Re: Tips for managing Perl projects?
by Tommy (Chaplain) on Dec 16, 2004 at 03:47 UTC
      Why use subversion when there's much more powerfull tool written in perl?

      Ever heard of svk? http://svk.elixus.org, svk, it uses svn backend, but it enables interoperating with other versioning systems like cvs or perforce or even subversion

      svk is an excelent showcase of perl prowess and it's ability to work with different existing repositories makes it easy to recommend to someone ( especially perl programmers ), because you're giving them just a great tool, not forcing them to 'upgrade'.

      "Oh, you're using cvs? you shouldn't, move to svn" - yeah, right, and who's going to move existing repositories and THEN tell all other programmers that they should abandon cvs AND tools that use it ( like jDeveloper, go tell them that they should really be using vim and svn ).

      With svk you can just use it yourself and it will push your patches/changesets to whatever your team is using (well, almost whatever, it will work with preforce, but won't work with highly proprietiary tools like bitkeeper)

        svk looks promising, but personally, I don't think the documentation is there, yet. Whereas with subversion, there's the fantastic Subversion Book. Download, print, read and go. Perhaps somewhere down the line I'll feel that svk is worth a second look. The OP was looking for guidelines on future projects -- from a fresh start, I'd second taking a good hard look at subversion.

        Regarding the migration challenge, the Subversion book includes an appendix on Subversion for CVS Users, which also discusses the latest tools for migrating a CVS directory to subversion.

        -xdg

        Code posted by xdg on PerlMonks is public domain. It has no warranties, express or implied. Posted code may not have been tested. Use at your own risk.

Re: Tips for managing Perl projects?
by zentara (Archbishop) on Dec 16, 2004 at 12:55 UTC
    Store all documentation and code in the RCS

    I'm no expert in big project management, but the first thought that came into my mind, was "have the programmers save ALL code they write", even failed snippets, in case you need to "prove" that you didn't steal some copywrited code, and came to find it independently". This in case someone at SCO ( :-) ) decides to file a frivilous lawsuit against your code.... or somebody taking your code, and saying it was theirs.

    When I start a little mini-project of my own, I generally have tens of "failed" attempts, each leading "step-by-step" to the final code. This could prove very useful to a legal department trying to mount a defense of it's "independent derivation".

    In the business world, it's more about legality and money, than programming itself.


    I'm not really a human, but I play one on earth. flash japh
Re: Tips for managing Perl projects?
by nothingmuch (Priest) on Dec 16, 2004 at 22:56 UTC
    IMHO, the two most important design principals go hand in hand.. These are minimalism (a subset of laziness, but also something entirely different (small, compact, decoupled bits)), and robustness (never cut corners. You should design so that the corners are left in tact, even if between them everything is just a stub). I use # FIXME comments a lot, when I don't want to lose my concentration, but I make sure that the places that have to be fixed are really problematic, and can't be worked around more eleganlty instead.

    If you design for these two, possibly with the latter being the tie breaker, you usually end up forcing yourself into writing concise, maintainable, reasonably efficient code.

    After that, optimise for readability, which implies robustness (due to less errors especially in maintaince), and interface flexibility (use the law of demeter, as well as tests (especially having to do with mocking) to try and find the weak points of your interfaces).

    These basic principals are easily enforced by unit testing as both a test for your design, since it is the first real world usage the code gets, and as a measure for robustness (Devel::Cover helps you find edge cases you may have overlooked, and code that shouldn't be there (which should be considered as more room for bugs)).

    I usually write two scripts per piece of functionality - an api test, and a real world test. A real world test makes use of as much of real examples as possible, and helps with the design, as well as overall sanity. It does one thing, usaully. The api tests cover every corner, and make extensive use of mocking/white box techniques. I never ever test more than one piece of functionality in a single test, because that confuses stuff. As time goes by regression test files for errors are made, or appended to the api tests, and the critical sections get stress tests, which is a deliberate attempt to try and break the system (usually these are worth a lot once, when you first write them, but not as much afterwords).

    Provided you keep things small, and tidy, and well behaved, and always listen to your gut if it tells you you're doing something you don't really like, you should be OK.

    Places where I get a bad gut feeling are, for example when the logic knows about special cases having to do with the environment, or when tests jump through hoops to get the job done, or any kind of repitition.

    Lastly I'd like to mention that the best sanity check is to get other peoople to help you think, usually about design. Code reviews are hard to get done because usually deadlines don't let them happen as much as they should, but design reviews are really a necessity. And if you can't get design reviews, then try and design as early as possible, but implement as late as possible, so that you can more easily criticise yourself about your design in retrospect.

    -nuffin
    zz zZ Z Z #!perl
Re: Tips for managing Perl projects?
by BravoTwoZero (Scribe) on Dec 17, 2004 at 14:38 UTC

    There's far too many good comments for me to add too much, but I'll try two anyway:

    1. In the U.S. and large enough? Think Sarbanes-Oxley compliance...
      Operationally, anything you do to touch access and account creation may be subject to SOX compliance. We're in the midst of it now. At them moment, they don't much care about how the code looks, but they do care about who can run utilities and where the output lives. Does it touch your financials or order-to-cash? Then, it's definitely a consideration.
    2. Print out gold versions for offsite storage...
      Sure, you're making digital copies for DR (right? ... the only right answer is yes). Print paper of major revisions, put a copy in the fire safe and take a copy home (assuming that's not a license violation... if it is, send a copy home with a company officer).

    I know these sound pretty dorky in comparison to the great "best practices" coding comments above. But, this frequently gets overlooked in project management. It's even worse when it's a critical admin utility. Make sure the code is auditable. Make sure there's a copy offsite.

    Amatuers discuss tactics. Professionals discuss logistics. And... my cat's breath smells like cat food.

      > Sure, you're making digital copies for DR (right? ... the only right answer is yes). Print paper of major revisions, put a copy in the fire safe and take a copy home (assuming that's not a license violation... if it is, send a copy home with a company officer).

      For code preservation and data recovery, I have these rules:

      1. Developers each have a "local" RC repository on their primary network resource (that is, home drive on Windows, /home/$USER/net on our Linux systems). All WIP is checked into that repository for every change. (Changes are demarked when the dev runs a test).
      2. All changes are checked into site-daily RC at EoD.
      3. Any change must be eventually checked into the site-dev RC, but must pass all unit-tests first. Unit tests are run automatically on checked-in code.
      4. All repositories are backed up daily. (The normal corp. process takes care of having off-site copies and rotations)
      5. Daily/Weekly RC snapshots are backed up to DVD-RW, and encrypted to the VP and project manager's keys. One copy goes home with the VP weekly, and one goes home with the project manager daily.
      Anything I appear to be missing?

      radiantmatrix
      require General::Disclaimer;
      s//2fde04abe76c036c9074586c1/; while(m/(.)/g){print substr(' ,JPacehklnorstu',hex($1),1)}

Re: Tips for managing Perl projects?
by gaal (Parson) on Dec 16, 2004 at 19:51 UTC
    All programmers on the team must read PM regularly.

    :-)

      I actually had this as an informal rule at one place I worked and the junior developer improved dramatically. My best moment as a mentor came when he turned to me and said "Why aren't we using XYZ module?" When I asked him where he'd heard of XYZ, he said "Oh, I read a post on PerlMonks by so-and-so who seems to know what he's talking about." My heart almost burst with pride!

      Being right, does not endow the right to be rude; politeness costs nothing.
      Being unknowing, is not the same as being stupid.
      Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
      Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://415138]
Approved by Corion
Front-paged by davidj
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others studying the Monastery: (6)
As of 2024-03-29 14:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found