Most people use statistics the way a drunkard uses a lamp post, more for support than illumination. — Mark Twain

In an interesting article, “Know the Difference Between Your Data and Your Metrics,” Jeff Bladt and Bob Filbin illustrate the crucial “difference between numbers and numbers that matter” with the example of a YouTube video appeal for used sporting equipment which resulted in:

• 1.5 million views; but only
• 8 donation pledges and
• zero actual donations.

They argue that “... all metrics are proxies for what ultimately matters ..., but some are better than others.” The better ones they call meaningful metrics, to be contrasted with vanity metrics (such as the 1.5 million YouTube views) which look impressive but have no business value.

I suppose the moral of all this is the usual one: the usefulness of a tool depends not just on its inherent quality, but, more importantly, on the skill of the person using it. A sharp knife is more useful to a surgeon than a blunt one, but more dangerous in the hands of an amateur. Software metrics are a tool for IT managers, useful (or harmful) according to the wisdom (or naïvety) with which they’re collected and interpreted. Bladt and Filbin conclude:

Organizations can’t control their data, but they do control what they care about. ... Good data scientists know that analyzing the data is the easy part. The hard part is deciding what data matters.

So, where does this leave the (non-management) programmer? What is he or she to do with all these metrics? Probably — not too much. In some cases, of course, the programmer can increase his performance by aiming to improve specific, targeted metrics. For example, a programmer who is considered “slow” might aim to increase his output by monitoring weekly LOC, challenging himself to produce more code each week. But this quickly becomes counter-productive if the percentage of bugs increases. In that case, he has merely increased quantity at the expense of quality.

My conclusion is that metrics are useful for management, much less so for programmers. The programmer should, IMHO, focus on the task of producing quality software; in other words, on nailing down requirements, honing the design, and writing clean, self-documenting, well-tested code. Produce good code, and the metrics will take care of themselves.

Thanks to eyepopslikeamosquito for an interesting meditation.

Hi, I have to applaud you for the effort of putting all this together, however, it seems to me that it all just makes life more complicated. I want simplicity. I don't want to worry about the efficacy of toilet paper. Maybe I'm getting old and don't want to learn new tricks, but your article presents alot of truth that would be useful to Edward Bernays' descendants.

All this agile thinking has made me stodgy and stiff. ;-)

As with many things, developer metrics can work well when chosen from the bottom (of the organization) and can be horrible when imposed from the top.

The only developer metrics we use are ones selected by our own team and those are only short-term for getting improvement on a specific problem. We haven't resorted to that recently. So that would be the Process Improvement part.

The main place I see pushing to identify metrics is around determining how successful a feature or product (or other "story") really is. But that seems tangential to what you were asking.

There are various rewards for developers and many of them have individual, team, business unit, and corporate-wide components. We have performance reviews and PKIs (using a different name).

We don't use metrics in performance review. Metrics might make them seem less subjective, but they'd mostly just make them more stupid. There is no way around needing a good manager. A desire to impose metrics on performance review is usually a reaction to some of the managers being worse at their job. The best metrics can't compete with even a moderately good manager. Instead of metrics, invest in improving how good your managers are.

We do lots of other things to try to reduce problematic subjectivity: have company-wide guidance on how reviews are done; do them all at the same time; incorporate feedback from top-down, bottom-up, laterally, and cross-team; separate performance review from compensation review; keep pushing to minimize as much of the process as practical.

Some PKIs have metrics, but each such would be chosen by the people closest to the particular PKI. So some of my personal goals have metrics but those metrics are meant to aid me and my manager being more objective in accessing progress on the goal and to improve clarity on expectations. Those metrics mostly don't matter in the PKI assessment process. My manager and I discuss how successful I was at each goal and assign a percentage grade to groups of goals for the purpose of determining bonus amount. We write up a very short assessment and we might mention numbers from some metrics, but what matters is the percentage score we agree on. Rarely there can be push-back from a higher layer of management but that is pretty rare because most of this is no surprise; even the President of the company has some idea of how well I'm doing before he sees my PKI paperwork (and that is true for most of the 200 employees).

The broader PKIs are more likely to have metrics or to be completely expressed as a particular metric (for example, there are usually a couple of direct financial metric goals). Those end up being less important and are just an attempt to encourage alignment at a broader scale and mostly have value, IMHO, when we meet those goals because it lets everybody share in the success.

But the real value in all of this is facilitating the flow of information and expertise such that each person has a clearer and more accurate picture of how well they themselves are doing and how they can do better (and are less likely to feel that others are "getting away with" doing a poor job).

Rewards are actually quite dismal at improving performance. Read "Punished by Rewards" if you want to know more about that.

This is an outstanding piece of work which I will refer back to. Thanks for this!

Having worked in small, loose organizations as well as large, highly structured organizations and some variations in between over the last 30+ years, I have found KPI's and other metrics primarily punitive. Human performance is subjective. Measuring a human as if they are a piece of equipment(in spec/out of spec, in tolerance/out of tolerance, effective/non-effective, efficient/non-efficient) is, to me, simply demeaning. Every one of those measurements is subjective when applied to a human.

I don't see this as a problem of measurement so much as compared to what we choose to measure. All kinds of methods are being and have been developed to measure the performance of employees, with the supposed goal of being fair and impartial. I have yet to see one which is not gamed, either by those applying the measurement or those who are measured. Show me any contrived system and I will define it as broken upon conception, because the one sure way to get ahead is to abuse a system, and thus a system will be abused.

One of the problems with any of the 'measurement' systems is that they attempt factor out the core value that humans bring to an endeavor: Diversity. Analogy: Measure every individual fruit in a fruit basket based on the same criteria. Measure apples<=>oranges<=>grapes<=>lemons<=>mango<=>etc..., all on a single scale. I don't care how intricate you make that scale, how you try to balance it. The bottom line will be that a subjective call will have to be made to accurately reflect 'value' brought by any given fruit. Try it with taste as the scale: I would venture to guess that lemon will come out below apple except for in unique cases that fall well outside of six-sigma. Then apply the scale of 'beneficial vitamin C', probably a similar result with a different winner.

Another challenge with these 'measurements' is that the 'environment' around that which your are measuring constantly changes. A good example is when contractor(s) are brought it to assist with a project. In the fruit bowl analogy, this is adding nuts to the basket of fruit. I suppose that it could be argued that nuts are a fruit, but regardless, when they are added - and subsequently removed - the measurements generally do not reflect that change. Again, the subjective is required in order to compensate for the skew not captured in the measurement scale.

That is two examples of what I have seen which convinced me that the quarterly performance review was a worthless venture(except in the rare case where I have had a manager who pencil whipped the thing with me, and then spent the next 45+ minutes discussing opportunities, problems and solutions, and generally giving me a better overview of the organizational direction and mode of operation employed to achieve goals). There are more examples, some of which are referred to in the work by eyepopslikeamosquito. The fact that there is "So Much" in that posting goes to my point from my perspective.

Productivity is not a check-list process. It is an art, not a science. An employee, regardless of whether they are the measured or the applier of measures, would be wise to examine and reflect on that list of items under the 'Agile' heading in the post. There is nothing inherently wrong with that list or the any of the individual points within it. However, to implement it as a check-list process(as I have experienced in most cases) is about as demotivating as one can get. As I generally believe that a system can't work unless all involved in the system are working within it, I have tried to work within the constraints of the systems, and generally lost because of it. Those that ride the fringes and step outside it, and are given the lee-way to do so, generally gain. Maybe I am just a loose cannon..., but I tend to ignore that which I see as broken due to complexity. And measuring human performance is as complex as it gets in my view.

One last point before I conclude(and this may come of sounding political again. It is laymen's philosophy in my view, not political): Human endeavor is a process, not a result. A given result usually involves orders of magnitude more process than we give credit for. We plan to achieve a result, then we endeavor to put in place that which will allow us to achieve the result. The result is given such an intense focus that we tend to forget the importance of process. By that I mean the tangential results of having endured the process. Those are the things that these measurements never really address. And those are the very things that bring long lasting value to the next endeavor, value to the laying out of the next plan, value to the teams and individuals that make up those teams in terms of skills and wisdom. We are in such a hurry to get that damn result, that we tend to throw out the value of having achieved it. So much effort in coming up with what to think as compared to taking the time to examine how to think...

To conclude: I do not mean in any way to take away or diminish the work or the content of the post I am responding to. As I stated up front, it is excellent and useful. In many ways, this is an unfair response:

• The original post is well researched, my response is anecdotal.
• The original post is well laid out and organized, my response is somewhat slapdash and probably incongruous.
• The original post is referenced on existing works of others, my response is based on nothing but my opinion and personal experience.
• The original post reflects some experience in a management role, I have avoided that role like the plague though have managed teams informally as part of projects I have been on.
• The post approaches the issue(s) openly, whereas my approach is pretty narrow(probably with a bit of attitude tossed in)

I appreciate the post and the opportunity provided by this community to present my thoughts, regardless of the value or lack thereof(of my thoughts)... :-)

I have two takes on the premise of the use of metrics to measure, or regulate, or reward the writing of code.

Agile -- simplistically stated -- uses the number, or rather the percentage of, tests passed as its primary working metric. And that encourages producing lots of, simple, easily passed, pointless tests.

You doubt this? Inspect *any* perl module that uses the Test::* conglomeration and see if it uses require_ok( 'Some::Module' );. If it does -- and most do -- you've seen this encouragement in action!

This is a completely pointless test.

If:

• (this instance of) Perl is incapable of loading a module:

  then the module (Test::More?) that provides the code for require_ok() will not be loaded and the test will never be reached.

• The module (Some::Module above) has not been installed:

  The the test suite containing this test will also not be installed, so the test will never be run.

However, if the module has been installed; but the Some::Module file has been 'emptied' except for the return 1; (or equivalent), then the test will be run and pass; despite that no other test will pass as no actual code has been loaded.

The upshot of that is that in the only scenario in which the test actually tells you anything (the latter case), it gives a false positive resulting in a 1/N tests pass instead of a 100% failure.

Note: There is no "gaming the system" here. The tool(s) provide the facility, the synopsis and detail promote its use; the users simply use it because "it seems like the right thing to do". The fault lies with both the tool -- for the provision -- and the philosophy for promoting the use.

My second take is that the moment you try to separate the physical construction of code -- kloc, function points, abstracts test quantities -- from the intellectual processes of gathering requirements; understanding work-patterns and flows; and imagining suitable, appropriate, workable algorithms to meet them; you do not have sufficient understanding of the process involved in code development to be making decisions about it.

You might just as well employ a mortuary attendant to run A&E triage.

Sussman attains enlightenment reads:

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened.

As with artificial intelligence, there is little point presupposing what a real intelligence will or won't think. To know what a person is thinking and what motivates them, it is helpful to ask. Then, observe their actions to see where the words and actions hold together or where they take different paths.

If management rewards lines of code for the sake of lines of code, then the programmers will gladly produce the lines of code the management requested. This is not the programmers gaming the system. This is the programmers producing what is asked of them.

If the management asks for high bug fix rates, they will be provided buggy software with lots of bugs to fix. This is not the programmers cheating the metric. This is the programmers giving management what management requested.

If what the management really values is small, maintainable, minimally buggy code then management should damn well ask for small, maintainable, minimally buggy code. Then management should allow developer and tester time to the task of designing, developing, refactoring, reviewing, and testing code. Make sure the developers and testers feel safe to criticize one another's work and more importantly feel safe to have their work criticized.

Let me take a particular agile framework as an example. I feel the reason Scrum works so well has less than the progenitors of it think to do with the regularity or exact length of the meetings, although touching base together briefly and regularly is a great thing. I think the biggest factors they got right are the self-managing team that answers as a single unit and having that unit submit their work to a single vision. A good product owner who can stand in the gap between the customer and the development team, and who accepts the work of the team to present to the customer for their acceptance is the secret ingredient in the sauce.

I've seen some complaining around the net about Scrum as having a high overhead. Well, that's true compared to some small teams but it buys a great deal of value for that overhead. One needs to communicate frequently to be truly agile and adjust frequently. One needs to have an at least partially fixed spec for a while to produce something of value without confusion, even if that spec is a bit nebulous and only stays fixed this two-week iteration. The team needs to lean on one another and mentor one another to prevent fortresses of individual skill and knowledge. Those fortresses fall hard when the team is realigned, and can bring down projects or even companies. It is best to prevent them.

Agile development is about getting the minimum work done to meet a quality and completeness goal as quickly as possible. There's a maxim we should all know by now: "Fast, cheap, good: pick two". There's no way to get good software fast without a little overhead cost. Scrum I think balances the cost to the benefit pretty well.

/me nods ...

“You’re over-thinking this thing,” he muttered, thinking about the book, Cheaper By The Dozen.   (A hilarious book by the daughter of an early process-management consultant ...)

Fundamentally, the writing of computer software has to do only with the success or the failure of the computer software to fulfill the purpose for which it was written.   Infrequently, the cause is programmer incompetence.   Usually, it’s because people are not-working together on a project that has never been completely specified in the first place.   And they’re not continually testing what they have done, to know if it actually works and keeps working.   In other words, yellow-stickies or not, it’s hopeless.   Whether or not you are measuring “progress,” the project’s taking one step forward and two steps back.   You measure time that is being profoundly wasted.

The best idea I ever had, a couple decades ago now, was to base things on a contractually binding task-order and change-order system ... and to make task-order #1 (and beyond) consist of building the remaining task breakdown.   Charging the same money to do that, or more, than for building code.   It’s modeled after the construction trades, and it works.   If you systematically embrace all of the best-practices that are involved (start to finish ...) in constructing a building or a roadway, and apply these to software ... it works like nothing else does.   (A project to repair the potholes in 100 feet of bad-condition street might consist of more than 250 line-items and accumulate the data points of more than 1,000 tests.   They leave zero, nada, nothing, zilch to chance or “creativity.”)

The usual practice ... and Agile encourages it ... is for people to be “lone wolves, in a pack.”   They forget that they are building a machine, that will work entirely without human intervention.(*)   The process they use is the same one they used in school ... at 11:30 PM the night before the assignment was due.   They’re frankly entirely accustomed to “pantsing it,” with only a token nod to planning, and no more of a nod to communication than will fit on a 2" x 4" yellow piece of paper.   Almost invariably, one person will be tasked with devising and writing the code for some feature, and for telling the rest of the “team” what s/he did after the fact.   I don’t know why this is so, but I’ve seen it in far too many teams to think of it as accidental.

So, what you really need isn’t “metrics” at all, in the sense of lines-of-code, hours-worked and what-not ... except possibly for cost accounting.   You don’t need to measure what your programmers are doing.   Instead, you need to be checking-off their progress along a meticulously-prepared punch list, cleanly separating the process of “deciding what to do and when to do it” from “what the source-code writers are doing.”   It doesn’t matter if folks are standing up or sitting down.   If the plan is good and complete, the work will go according to plan.   (And, yes, that plan should called for staged delivery and for overlapping threads of progress.)   If the work has been spelled out sufficiently, the writing and testing of the corresponding source code is ... perfunctory.   And the resulting code will be damned near bulletproof.   If you measure anything at all, it should be the on- or off-schedule progress of the machine that is being constructed day-by-day.

_{(*) The best book that spells out that idea very completely, from a project-management point of view, is a little Kindle-only book called Managing the Mechanism, by Vincent North, which is available [only ...] on Amazon.   Find it, read it.}

Management use individual KPIs to reward high performers in Sales and other departments. They are contemplating doing the same for Software Developers.

They've tried this before in the 1990s, it didn't work well. There are many variables, and a huge variable is: your vendor's modules/software doesn't do what it says it does. Keep in mind that marketing people, not software engineers, usually write the marketing material. Thus the marketing people really don't understand what a product does, and when programming gets involved, details matter. This was so bad that in one company I demanded to review all documentation written for changes I made. (We had a marketing person writing technical documentation. Sometimes it lacked important details, and sometimes it was just incorrect.)

Example: In the 1990s I was using MS Access. The box said it was backwards compatible with all versions. It was not. It only converted forms, queries, reports, and macros, not code. When a new version of Access was installed, all code had to be retested, and sometimes rewritten. If you used a lot of code, it was a sure bet something was going to break, especially if you used a DLL that got upgraded. When the OS or another product (like Access) was upgraded, some DLLs had their data type changed to another larger type, from integer to long for example.

My point: you really don't know if your vendor's software will do everything it says because you can't trust what's on the box or in the documentation. Good luck getting your vendor to fix something (that should have been designed correctly in the first place) just for you without paying through the nose.

Perl 5.8.8 on Redhat Linux RHEL 5.5.56 (64-bit)

A key problem with these metrics is that they come from the world of manufacturing.   In those lines of business, you are performing a repetitive process, with one iteration (and outcome) being independent of all the others, and the process is basically deterministic.   You can trace defects either to the worker, to the materials, or to the line.   Measurements of performance are meaningful and useful; incentives and bonuses might work.   However, none of this is true with software, and for a variety of reasons.

First and foremost, software systems are, as the book puts it, chock-full of “interlocking interdependencies.”   Everything is connected to everything else.   The flap of a butterfly’s wings over here does cause a hurricane over there.   (One of the key take-aways from test-driven development, and therefore one of the reasons to be using it even on your own solo projects, is to see just how easy it is to cause a seemingly-unrelated set of tests to start failing.   You won’t believe it, at first.)

Second, and as the book also mentions, the thing is a machine, which will operate entirely without human intervention and beyond human control.   No one will know or care whether the team that built it was standing up or sitting down, but they damm well will care if it does not work ... perfectly.   It is entirely possible, and frequently the case, that a large block of carefully-measured (and expensive) time will be a 100% sunk-cost.   It is also entirely possible that it will be impossible to predict “how long it will take” to solve a problem, since the part of the problem-iceberg that pokes its head above the water is only a minuscule portion of the actual thing which might be broken or defective ... or inadequately or incompletely designed.

Finally, there is the natural human tendency to want to say, “Naming it will Solve it.™”   Although some of the names we come up with are perfectly nonsensical ... (“Scrum PLOP?”   Really??   What sort of sh*t is that?) ... giving them names or TLA’s (Three-Letter Acronyms) implies that we actually know what we are talking about.   This quickly creates a gigantic credibility-gap.   (“Fool me twice == you’re fired.”)

Another excellent book on this subject is Hollywood Secrets of Project Management Success, by James Persse.   Dr. Persse, who by the way is an excellent and engaging author, was given the seriously-cool opportunity to get behind the scenes in the motion-picture business, which, as he points out, does have a demonstrated track-record of bringing multi-million dollar long-running projects to completion.   Their project management practices have nothing to do with things like “Agile” (although they do employ overlapping, staged delivery).   They seem to be quite inflexible, set in stone, although they aren’t.   It is a scrupulously-repeated multi-stage project life cycle that has been honed to perfection, because it works.   You can “bet a hundred million dollars” on a movie project and be certain that you will get a piece of celluloid, on time, to show for it ... which is a lot more than we can say for software.   (And, don’t say that the two types of projects are incomparable, because, being a software technologist himself, he quite convincingly demonstrates that they are.   There is a lot that we all can learn from this book, as well as from Mechanism.)

Quite honestly, my biggest beef with the Agile loudmouths (ahem ...) is that, while they certainly “talk a good camp,” they can’t show that their processes actually work.   (Which may be one reason why they talk so loud ...)   Yes, there are very good reasons to favor staged-delivery and iterative work cycles, when appropriate, but the actual process of creating “software Mechanisms” is vastly more intricate than their theorizing would allow, and the “fabled outcomes” simply don’t materialize.   Their premises, while filled with some pretty good ideas, are also insufficient.