... in the software development industries, we happen to produce huge amounts of it...

Your analysis neglects two pieces of information I've found relevant on my projects.

First, code which produces a return on investment today produces tangible value today, even if that code will become scrap in the future.

Second, code which helps me discover a better approach also produces tangible value.

I've written a lot of code I've suspected I would have to revise and rewrite in the future as the project took on more responsibilities. I am often right about needing to do this, but I am often subtly wrong about how the code will need to change until the point at which I absolutely must make the change.

Fortunately software is pure thought-stuff, divorced from the petty tyrannies of material economics, so discarding a thousand lines of code in a dozen files favor of a hundred lines in one file may not be a waste at all.

While I completely agree with you about the "investment today produces tangible value today", I must say that this is probably the most common error made by so-called agile teams.

Let me step back a bit and ellaborate...

A project (according to both PMI and Prince2 standards) is something that has:

• a start
• an end
• delivers a specific product.

It's a common misunderstanding that scrum (or xp, or whatever) is a "Project Management" methodology. This is simply not true, because in Scrum the so-called project doesn't have an end and it doesn't deliver a specific product. Scrum (or xp, or whatever) is a "Software Development Process". This might seem like a silly conceptual nitpicking, but it does make a huge difference.

Agile methodologies are very much useful for contracts of ongoing maintainance and evolution of a already running software, because in that case, the idea of "release soon, release often" is the most efficient way to deliver the most value-added to the customer. Keeping it in strict time-frames gives a better manageability of the integration, release, deployment cycles. Although this is often called that way, it is not, by any formal definition, a project. It's an ongoing process.

The basic practical difference between a "project" and an "ongoing process" is the management of the stakeholders and the expectation of the scope of the work. When you have an ongoing process, you should do now the work that adds value now. When you have a project, you have a "specific product" you must deliver, and it doesn't matter how much you try to transfer the role of "requirement analisys" to the customer, it is only going to get back to your face. In a project you really need to do the "big design up-front", otherwise you'll have too much scrap, and nobody wants to pay that.

And this is probably the hardest part to understand. Even if you want to embrace agile methodologies, you need to understand that the first release must be managed as a project, with a start, an end and a specific product to deliver. You need to do all the requirement analisys of this first release (I even recommend making an effort to turn it into the smallest possible project), but you can't do that first release in "sprints", because it doesn't make any sense precisely because you are not adding any value because the system is not in use yet.

So build your planning, assign the resources, keep room for fixing bugs and eventual misunderstanding of the requirements. And move to a agile methodology as soon as the system is up, running and successfully deployed. But never before that.

Rework is not always scrap, but rework when the system is not yet deployed is always scrap. Most failing Agile "projects" I've seen were failing because they forced agile in the "first release", and the customer simply refuses to pay for rework or to features misprioritization while the system is not yet deployed.

daniel

... rework when the system is not yet deployed is always scrap...

I question that because:

... keep room for fixing bugs...

... and:

In a project you really need to do the "big design up-front", otherwise you'll have too much scrap, and nobody wants to pay that.

"Just plan harder so you don't write bugs" is hardly realistic.

(If you define "scrap" as "stuff you have to throw away because it has provided no value", then you have a circular definition you can only break by redefining "value". You get to pick what you rework: is it the BDUF spec several months later or is it code minutes to days later?)

Even if you want to embrace agile methodologies, you need to understand that the first release must be managed as a project, with a start, an end and a specific product to deliver.

Every reputable agile method of which I am aware suggests this explicitly.

As with the exotic-metals machine shop, the things that really devastate a project are the things that wind up in the scrap bin.

I disagree. The things that really devastate a project are things that should wind up in the scrap bin, but don't.

That's why I think it's dangerous to measure the size of the scrap bin, and equate "big" with "bad". It encourages people to keep things in a project that don't belong there. In contrary to the metal shop the scrap/no scrap distinction is rather hard to, and can often be argued about at length.

There is a great deal of difference in that it is fairly straight forward to measure the efficiency and effectiveness of the processing of metal parts. Those processes can be broken into sub-processes and further into process steps. Each step can be measured, and so can the results of each step.

That is very different from coding, true. There are of course process steps in producing functional code. Measuring the effectiveness and efficiency of the process and the end product is harder to do because there is little that is tangible....in most cases.

But code that diddles bits, changing data into information is not the only code that gets written out there in the real world. The machine shop is a good example. CNC, DNC, PLC... all of these have to have code written for them. It is not nice code like Perl either. Ladder logic is a PITA to write. And code like that does directly impact tangible products in very serious ways. It is not just scrap product that gets made, but the product that gets made that we don't realize is scrap that is dangerous, and often due to poorly written code that comes from poor development processes, poor specifications, poor understanding of the interface between the code and the real world.

The analogy made between the machinists and the coders is a bit of a stretch when is comes to measuring effectiveness and efficiency. The very real as compared the very abstract. There is however some middle ground in there that does bridge the two. A manufacturing execution system is a good example. It spans the abstract world of business decision making and the very real world of machine control systems. And at every level there is a boatload of code that exists to do more than make graphs and charts and reports and stick them on the corporate intranet. That code keeps millions of dollars worth of machinery and a good number of folks working safely and making safe product.

The nebulous nature of code is both its strength and its weakness. We can change it often and quickly. Not so with a 5 ton punch press, which is nice in a way, because with a 5 ton punch press you always know precisely what you have, and are not likely to change it much.

Perhaps the measurement of effectiveness and efficiency needs to be described in such a way that it encompasses both worlds.....

I am not stating this very clearly(because it is 01:00 and my grandson is keeping me up), but I appreciate the folks that posted here for what they have written. Made me start thinking...

“we’ll meet for exactly fifteen minutes, then we’re going to spend the rest of the day cutting metal, because cutting metal is what we do!”

Perhaps the “scrap rate” is the actual metric that we should be infatuating about. And we might be able to obtain such a metric right from our version-control system logs. Look at the total “surface area” of a source-file that gets changed over time. Particularly observe when a large chunk of code is first added, then either replaced or extensively modified a short time later. If there is a defect-tracking system, try to correlate this “source-code volatility metric” with those defects. Do the same thing with the activity that is being generated by the “scrums.”

Once again, I want to remind the reader I'm not advocating scrum. But I think it's being misrepresented, and I don't think the "scrap metal" analogy is valid.

Bits aren't a precious commodity. Code is cheap. It doesn't involve tons of rock and dirt to be moved and crushed to extract a few fragments of ore. A small error doesn't mean a days work is scrapped and we'll have to start all over again. Even code that is written, tested, and then discarded has value: we have learned something (this feature isn't "working" - where "working" typically means "making money for the company")

It is truly impossible for me to dis-agree with that comment more vociferously than I do.   “Bits,” as you call them, are in fact one of the most-expensive work products that exist.   And sloppy business practices are one of the big reasons why.

What would you think about a building contractor who showed up with his crew and started furiously cutting boards and nailing them up and, about half an hour later, the foreman walked over to you and said, “now, what exactly do you want to build here?”

Let’s say that you do not grab the nearest two-by-four and club the guy, as he so richly deserves to be clubbed.   Let’s say, instead, that you play along (“he’s an expert, after all”), and so you tell him.   And so he tells his men to rip down all the stuff they’re building and to put up something else, instead.   Which they promptly do, with just as much energy as before.   The foreman points at the pile of broken wood and says, “but it has value.   It was a good learning experience.   My boys just learned something...”

You suddenly put two-and-two together.   He’s expecting you to pay for all of that wasted wood!

Are you grabbing for that two-by-four now?   You should be.   And yet, here we are, telling upper management types that they should be paying whatever bill we send them, “because we are obviously so busy.”

I can tell you a business practice that works, because I have done it for years:

• You will start by signing an eight-page contract.
• Everything that is done, is done under the auspices of a “task order” which is part of the contract.
• “Task #1” consists of “creating Tasks #2-n.”   You will pay for this service.   In fact, you should not be alarmed if several specification-tasks are conducted, possibly consuming one-third to one-half of the project’s contemplated costs, before any code is written by anyone anywhere.
• A definite price will be attached to each task-order; 50% down, 50% on completion.   Acceptance of one task order does not obligate acceptance of any other.   The terms must be mutually acceptable to both parties.
• If you change your mind ... that’s a new task-order that amends the first, and we don’t have to consent to it.
• You are interested in the results obtained, not where the work is done or exactly how the work is done.   “Results will be obtained” only after (you pay for...) a very rigorous definition of what those “results” shall be.   And you will do that, and we will compel you to do that, because . . .
• The completed work is guaranteed against defects in workmanship for a period of 30 days following delivery.   If any defect against the stated task-order specifications is found, you do not pay for the direct costs of the remedy.

I contract for all sorts of services under similar terms put forth by the professionals that I deal with.   In fact, I really do not choose to deal with anyone who does not pursue their business with such rigor.   I have waited in line for weeks to have work done by those trustworthy contractors, and I will continue to do so.   I have a careful file of business cards, and plenty of extras.

The expectations of internal and external business customers are no different.   A single computer programmer, within a team of dozens, might represent a capital outlay of several thousand US Dollars a week, as that person pursues a mission critical undertaking.   “Code is cheap?!?!”

What would you think about a building contractor...

Having worked in both construction and software, I can tell you that construction is more like software than software should be like some mythical view of construction that doesn't actually work in practice, either.

Q: What do you call a construction project where the crew shows up on the first Monday morning at 7 am to a two-inch pile of change requests? A: Normal.

I have read your SCRUM articles with great interest. I myself was once trained in eXtreme Programming but more recently every company has to SCRUM (another silver bullet?). So I Enjoyed your articles very much and it got me thinking as well:) However, what you wrote in you post is, IMHO, wrong.

Let me give an example. We developed a prototype to elicit and validate the requirements. Now that we know what we want we throw it away and redevelop it properly. We are happy to throw it away, although it sort of works and the look-and-feel is OK, the quality of the SW is very poor. Management on the other hand does not appreciate this, why throw something away that works? In fact if we would not throw it away, let's assume we were forced to keep it, we would have areal problem. It's basically exactly as moritz++ describes in his post: "The things that really devastate a project are things that should wind up in the scrap bin, but don't."

I don't think the "scrap rate" metric is very useful, maybe for some specific types of projects. If you do a lot of refactoring the scrap rate will probably be high and that's perfectly OK for many projects.

Cheers

Harry

I have read your SCRUM articles with great interest.

Which articles are you referring to? Just in case you're referring to the Nobody Expects the Agile Imposition series, please compare the author names. :)

You're very right, apologies, also to brother eyepopslikeamosquito.

To my way of thinking, “management’s” perspective on this situation is pefectly defensible ... even to the exclusion of all other possible perspectives.

From “management’s perspective,” it does indeed seem that they have just paid you whatever-it-is your salary happens to be.   And now, what exactly do they actually have to show for their hard-earned money?   It would seem that you are saying ... “100% sunk cost!” ... and, now you are dishing out to me what certainly appears to be very-lame apologies for a double-helping of thoroughly unacceptable excuses for blatant incompetence.   (Ahem...)

Please explain to me, once again, why I must pay a competent building contractor to construct a rather-routine building twice, in order for said “competent” building contractor to figure-out how to correctly build such a building once? . . .

Are you sincerely telling me that, at this late hour, I must fund and pay for a building-construction school in order to timely obtain for myself (and my clients...) a competently-constructed rather-ordinary building?

“I know how to budget for $1,000,000.00 worth of useful improvements to a city street, and actually get them.”   But now, it certainly seems to me that you are telling me that I must invest $1,000,000.00 in experiments ... just to “figure out how to” (throw away) $1,000,000.00 more...

“Pardon me if I am not impressed.”

Please explain to me, once again, why I must pay a competent building contractor to construct a rather-routine building twice, in order for said “competent” building contractor to figure-out how to correctly build such a building once? . . .

What makes you think software development is anything like construction? Given that I can duplicate the equivalent of a building in the software world with the cp command until my fingers get tired, the economics of software and construction seem rather different.

...we happen to produce huge amounts of it, apparently with no accounting at all. Maybe that is part of our problem.

What problem?

Software development is one of the few industries that allows you to charge for producing (s)crap, sometimes even for delivering crap and then charging once more for fixing it. Hardly any other domain lets you get away with such poor behavior.  ;)

I wrote a lot of what has been tottering on the edge of "scrap" on my own initiative in the last two years at work. The scrap in the form of unused application stubs and experimental deployment changes to modernize a legacy codebase and tests to do the same. The rest of the team is becoming interested in doing things this way. The new tests, though not formally or automatically run, have made some major changes much smoother and faster in a couple of cases for the devs and testers who were interested enough to learn and apply them.

So what could have ended as scrap is now a prescient investment. If there had been a rigid process about how I spend time at work this would have never happened and we'd still be squarely in the Perl 5.4 era with no hope of climbing out.

Thompson's rule for first-time telescope makers: "It is faster to make a four-inch mirror, then a six-inch mirror, than to make a six-inch mirror."

- Programming Pearls, Communications of the ACM, Sept. 1985

We're not just cabinet makers, we're also tool makers. What seems like a piece of scrap might end up being the only socket that will fit a bolt only hinted of at the starting blocks. I learn constantly in this work. Throwaway prototypes are often the fastest path to gaining expertise enough to not make a dog's breakfast of the actual solution. There may be scrap on the pile but the imprint of what it is, what it took, what it might be able to do remains.

This doesn't mean I disagree with what you're saying. I liked a lot of it and found some of it sincerely rallying. I think you over simplified and conflated and a lot of opposing ideas in these discussions though. Process here, in software development, is not simple.

Related: Thinking about Thinking (mappers and packers).

I know, I know, just when you thought you were out, they pull you back in.

I've been thinking along those lines: "Its not scrap... its a concept car and that clay model"

Surely that exotic metals machine shop did a test run with cheap dummy material and then threw it in the scrap bin before committing a whole lot of decimal places worth of the real materials to that process, right?

Isn't that why we have rapid prototype machines? Spit out a plastic widget and make sure it fits and works as expected before even thinking about spending 6 digits on tooling the steel parts.

One problem that I've noticed happens a lot, is that in software that SHOULD be pre-planned scrap turns into the final product anyways. Who is willing to spend the time to make a practice run, put the prototype in a wind tunnel, and then go back and rewrite the real application with experience.

When hardware is involved, nobody bats an eye at mockups and functional prototypes that are meant to be thrown out, and actually ARE disposed of.

/me nods...

Obviously in the engineering worlds there are places for concept-cars and prototypes.   It is perfectly valid, I think, for a group to say, “We are going to explore this strategy.”   And if we are doing it, knowing that we are doing it, and with a strategy in place for when we will be finished with that exploration and how we will evaluate its outcome, then ... that really is what we are doing.   But it is easy to try to use this appealing notion to excuse the desire to be “going somewhere” when someone does not, in fact, know where they are going.

I do not mean to reject what you are saying.   But how often does what we are doing really demand a concept-car?   Very few of us are “boldly going where no man has gone before” any longer.   Millions of commercial applications have been constructed by now, and we are merely building another one.

If you want to change your house, an architect will draw and re-draw plans for you just as many times as you will pay for the service.   But a professional housebuilder won’t proceed at all without those plans, and he will build what is planned (carefully cataloging all those change orders, and refusing any which represent a material change to the design, which must come from a licensed architect).   Yet, the the things that we do are many times more costly than a house.   Where is the engineering discipline in what we do, and why do we hear folks insisting that we do not need it?   We are not making magic.   And, we are not playing professional sports.

/me nods...

You’re right... it is not simple.   And the exploratory ventures that you made, I would say, indeed are not “scrap,” because you knew what you were doing and chose to do it in a certain way ... to better ascertain whether it would be a good idea to move the project in a certain direction.   And it seems that your intuitions were, as you say, prescient.

You are right, also, that I oversimplified ... because when one is making a point, and trying to do so in some semi-readable way, a single key idea needs to be presented clearly.   I hope that it did that.

If we can just get away from the idea that “the world of business is like a sports team,” that will be a start (although I doubt it will ever happen).   Folks love to watch sports, for some reason, and it never seriously occurs to them that what they are witnesing is, in a real way, a staged theatrical performance.   It isn’t the real world at all.

The reason for a lot of Agile code/practices is pretty sensible to me. IMO, the trouble is the language and process sometimes seem too dogmatic. I suppose it's a necessary evil, as creating common sense names for a specific practice that's intended to avoid legacy pitfalls that are too numerous to name, probably don't exist--so why not give them whacky names? If there aren't too many, I might even remember what they're for...

I've attended training on Agile, and it's funny to hear "experts" argue about problems and solutions that are found within the process. Often, like any dogma, it's left up to religious interpretation. There are faithful followers, zealots, agnostics and blasphemers. And then there's always some heretic that throws in his own set of scriptures or accesses some deep magic, and the conversations are all very strange.

But then, I suppose it's the price we pay for trying to avoid all the pitfalls of the past.

I've found a good general rule is to think of all these things as a way to help you stay engaged, while becoming better at what you do. If you're continuing to learn, you're doing something right. If you're doing process for the sake of process--then automate it or find a better way.

Ooooo! Ouch! You just stomped on one of my hot-buttons!

"But then, I suppose it's the price we pay for trying to avoid all the pitfalls of the past."

<rant> Alternate interpretation: It's one of the prices we (where "we" reflects my predominantly US-ian experience) pay for our restless search for a silver bullet; for a man on a horse; and for our predilection to prefer following dogma to exercising reason.</rant>

I see nothing wrong with creating more pitfalls, to avoid the old ones. It keeps us all busy. buwahaha.

The design of the part and the engineering of the process to produce the part were probably not the product of the machinist producing the part. Think of the machinist, no offense to machinists here, as the compiler. The mill, lathe, drill press, welder, and other machines he runs you can think of as libraries.

The software design and development process is fundamentally more like the design of the part and the process engineering for how to produce the part than the actual production of the part. The engineers select the metal, draw the plans, determine the steps of the machining process, and minimize the cut-away (also referred to as "scrap", but very different from scrapped product).

When there is a scrap product from a solid process, it's because there was a mistake made in the machining process or a defect in the raw material. Neither of these is the direct fault of the design engineer or the process engineer. Those scrapped parts may be blamed on the machines, the machinists, the machine maintenance crew, the materials buyer, or something else.

When there's a bug in the process, that's when you blame the process engineer. When there's a bug in the part design, that's when you blame the design engineer. Maybe some independent CAD drafter or CAM translation crew who cut corners in assisting the design engineer could be blamed. Most DE's do their own CAD drawing and CAM translations these days from what I understand. Some CAM and CNC software actually reads straight AutoCAD or Microstation files these days, too.

One of the biggest problems I see with software is the lack of a design engineer in the first place. We have plenty of tools and toolmakers to help us implement. We have plenty of advice on process. What is usually lacking is proper design.

You can't develop a process and implement a completely serviceable product without developing a solid idea of what the process is supposed to produce. This is where solid customer stories come in for custom software or software as a service. It's where good market and user research comes in for software as a product. In either case, a UI designer or usability expert can probably help. Someone skilled in writing in the problem domain probably can, too. I've actually found it helpful to have one or a few typical end-users assigned to connect with the development team to answer questions and try out every iteration of the software. It works much better for quality control than throwing code over the wall and taking back reports from "the customer" as a black-box entity.

Maybe you can find a good internal or external customer to pay for your software to be developed for them and allow you to sell the finished piece as a product. In that case you can do agile development using customer stories even for a shrink-wrapped product. Your customer's needs are probably not unique after all. If nothing proprietary is tied up in the code they might take a discount on the up-front work in exchange for letting you keep copyright and resell the work. Most of the agile processes work best when you already have a customer with a defined need, after all.

Overall a great discussion, thank you all. The well developed discussion of the differences of opinion reminds me of many battles I have been forced into over concepts such as Software Engineering, Development Methodologies, Management Quality Initiatives, ad nasuem. When you get right down to it the process of creating functional software is still more art than science or even engineering as it is understood now. Creating software is really more akin to the development of a commissioned art work than designing a new electrical circuit.

Why? Well I suspect that centuries or at least many decades of prior ‘art’ in the other ‘engineering’ disciplines has a lot to do with it. Software development is till something that is only about 2 generations old. Many of us here started our careers in the very early years of our art. We grow closer each year to becoming a true engineering discipline. But the progress is slow

I remember mangers who 15 years ago insisted that network services should be as robust as telephone networks. They forgot that telephony had had over 100 years to build that robustness.

Too many mangers still insist that methodologies that work in other industries and processes are perfect fits to software development. Six Sigma worked wonders for Fed-Ex and the Army Logistics command and ….. etc. so therefore it must be perfect to apply to the process of software development. Um not so much.

It I so easy to be drawn into the cult of measurement that we forget that we have no idea what we are trying to measure or what the numbers we come up with actually mean (if anything) … remember source code line counting?

We need quality processes to improve our products but I think that until we develop these things from with our own ranks we are not going to move forward very successfully.

We all have suffered managers who do not understand what we do, how we do it or anything about the process. I personally think that managers who are responsible for managing developers and other software folks should have been software people at some point. The numbers of us who become managers is growing but the percentage is still pretty low. .

As always these are my 2 kopeks worth. YMMV and probably will 8^). I do hope it will stimulate more such excellent discussion.

Misha/Michael - Russian student, grognard, bemused observer of humanity and self professed programmer with delusions of relevance

If you have never read the book, Cheaper By The Dozen, I would recommend it to you all.   This book was written by the daughter of one of the original management-practices gurus:   the people who counted the number of movements made by a person’s hands, to try to reduce wasted motion.   The kind of person who weighed a tube of toothpaste.   So, absurdity masquerading as science is not altogther new.

It is strange to have been on both sides of that manager’s desk, and I certainly know which side of it I prefer.   In the manager-seat, you are responsible for the outcome but you feel that you have no real control over it.   But then, when you turn away from The Dark Side   ;-) ... you find that you just can’t stomach the various “methodologies” any more.   You realize that they consist (IMHO... just IMHO...) of excuses for an utter lack of process, masquerading as process!

Yes, we can determine how long a project is going to take to complete.   We can bring the ship into the dock without the slightest bumping into it.   We can finish a plumbing job and turn the water on and not need to have a bucket and a wad of towels in our hands, “just in case.”

Maybe this is a natural (good) evolutionary process, but ever noticed how those that spend huge amounts of time upfront overdocumenting their designs, tracking effort, and writing long specifications for interfaces end up in management--and lose their ability to code fairly quickly?

Maybe that's just part of another trade-off? Programming competence requires programming--all the time. Filling out Design change requests and forms for peer review will probably get you somewhere too...

(Of course that's an egocentric view, unrelated to a customer who couldn't care less as long as he gets what he wants...)

I’ll once again very politely dissent ... and yeah, I won’t bother to keep doing so.   The perception that is obviously encouraged by many folks, among many other folks who ought to know better, is that ... if scrap code is not “free,” at least it is “cheap.”   It is neither.

Just let the record show that I only want to hire the programmer who damm well knows what kind of join to use, and who does not have to “try them out first.”   She is the one who I want to be working on a production dataset that I am right now looking at, which has 338,207,496 rows in it.   I am not running a school here.   I am here to try to figure out why another client’s project is crashing and burning, and, once again, I think I know.

I almost never find incompetence.   I usually find neatly indented code and I usually find comments!   What I always find is a process problem.   The team knows how to do it.   But, the team does not know what to do.   How can you “cross that bridge when we get there,” if there is no assurance that the bridge will exist, much less be capable of supporting the load (which is also “to be determined”)...   Fifteen years ago, I thought what I was seeing was an anomaly.   But I see it to this day.

If there is any central theme to what I am saying, it is that our industry has a reputation for “trial and error,” “fly by the seat of our pants (and we’re proud of it ’cuz what we’re doing is so magical and wunnerful that it has to be this way)” practices.   The sooner we can get rid of these notions and start doing what every other engineer or homebuilder has been doing for generations, the better off we’ll all be.   “Flyin’ away on a wing and a prayer” belongs as the theme-song of Greatest American Hero, but nowhere else.   This thing that we are all doing is not so gosh-darned remarkable; at least, not anymore.


(“Okay, I’m done,” he says. Quietly steps off the podium, picks it up, puts it away.)

That sounds more like a unclear or misunderstood requirements issue to me. But I may be missing your point.

I have worked too many projects where the ongoing answers from client and management were I am not sure what I want .. but that's not it Individual developers should never have to deal with requirements which leave them unlcear what they are supposed to do. And they should have some process to allow them to push back if they find themselves in such a situation

On the other hand a lot of the code I have written has been at least to me and the teams I worked with, relatively new territory with very little available prior art to build on ... not much you can do there but trial and error somtimes. Or cases where the delivered specs for somthing you had to interface with were just flat wrong in one or many critical points.

This may explain why I am a firm beleiver in an interative development process 8^)

Misha/Michael - Russian student, grognard, bemused observer of humanity and self professed programmer with delusions of relevance

I only want to hire the programmer who ... knows what kind of join to use, and who does not have to “try them out first.”

Riddle me this: what's faster, bubble sort or quicksort?

Nota bene that I have not explained the characteristics of the data set.

I must say that you are very fortunate to have talented developers, but that is not always the case. Luckily I am in a position now where I am a one-man army (has it's ups and downs, for sure, but at least I'm solely responsible for f-ups). In the past I was with a company where there was one person for each phase of SDLC. I lost count of the number of times where the Design document and/or Programming Specifications were completely useless documents written by someone who only had an ephemeral view of the intricacies of programming, and ended up being worthless, and a waste of time. Time that could have been put to better use.

I used the machine-shop analogy somewhat loosely, because obviously in the world of CNC there is a huge amount of “software development” of an entirely different kind.   (Yet it is subjected to very rigorous methods because it does translate directly into what is done to precious industrial metals.)

P.S.   There is never really any question when a machined part goes to the scrap-bin.   If it is off by a “tenth” (1/10,000 of an inch) in a crucial operation, the whole thing can be scrap.   And the computer software which we write might well be subject to equally-stringent standards.   Or, more so.   (If you don’t believe me, just utter the phrase, “patient health and safety issue,” in a medical-software setting.   It’s roughly equivalent to shouting “Fire!!” in a crowded theatre.)

Still, the essential idea is that of “scrap.”   And, the fact that we fairly spew it out.   Sometimes, I wonder if we mistake activity for progress.   Are we ever intending to be “finished?”   Are these processes really what our customers and employers want and need?   Or could we be more (self-)satisfied about them than they are given good reason to be?   Introspection never hurts...   we should be asking ourselves the hard questions.

And I wonder when, I’ll be home again, and the morning answers:   “Never.”
And the evening sighs, and the steely Russian skies go on forever.
– Al Stewart; Road to Moscow.

I believe, you've just fallen into an error of wrong analogy. One shouldn't draw parallels between completely incompatible things. The work with precious metals in your picture is done with the process completely established. Writing of software is establishing the process. Here you should compare it with the research labs of that factory that spend their time on deciding how to process the metal. I'm sure, they have lots of "scrap" before they come up with the process.

I will never, ever, accept the notion that most of what we do in commercial software-development actually consists of “boldly going where no man has gone before,” such that we have to construct lots of code ... only to throw it away and start over ... just as a method for figuring out what we are supposed to do.

Obviously, I meant the machine-shop as a parable, as an illustration, not strictly as an analogy.   In this and in several other ways, I meant to level a large cannon at the notion that “writing of software is establishing the process.”   It almost never is.   We have done most of the things that we are doing, thousands of times before.   We have smoked the mojo of one pundit after another who assures us, in the final analysis, that what we are doing is absolutely inevitable because what we are doing is so hard or so groundbreaking or whatever.

People “make things up as they go along” because, for example, they don’t perceive any value in planning what a database table is going to contain before they “just go ahead and build the damm thing.”   It seems like an extra step...

“Why spend all that time with blueprints when you can just start sawing wood and nailing it in place?   I mean, that’s all you’re actually going to do, anyway ... so why not just go ahead and do it?   Why waste all that time drawing all those stupid pictures? ...”

I worked on a Perl app for a couple of months, then wrote all the code in a week.   It was a matter of going page by page and writing each routine as-specified.   And, mind you, it felt a bit strange not to be “figuring it out and...”   But the code deployed, was used frenetically by lots of people almost at once, and it never had a single problem report logged against it.   It works.

During that process, of writing that document, I was programming.   There were several revisions, HTML mock-prototypes for screen layout, and lots of “discoveries” that would have represented a lot of code-rework had I been writing code at the time.   But I wasn’t, and that made a huge difference.   I applied the same principles to other team-projects for clients.   Very, very consistently:   the more work you put in up-front, before you write a single line of anything, the bigger the payoff will be.   The experience permanently changed my outlook, in a very positive way.   I know that it enabled me to study the business-problem more thoroughly.   It let me select the best tool for a particular piece, not merely the first one that I stumbled upon.   I was not “learning as I went.”   First, I learned; then, I went.   And, when I went, I went directly to the destination.   The little light went on, and it never went out.

I worked on a Perl app for a couple of months, then wrote all the code in a week. It was a matter of going page by page and writing each routine as-specified. And, mind you, it felt a bit strange not to be “figuring it out and...” But the code deployed, was used frenetically by lots of people almost at once, and it never had a single problem report logged against it. It works.

Good for you.

Too bad you just lack the imagination that if X works for you, Y actually may work for someone else (with Y being "different developers", "different company", "different product", "different customers"). I just don't buy the "X works for me, so Y must suck".

My customers are saying "XYZ may become big - can you do a small, no bells-or-frills implementation we can play around with"? Half of those pilot projects are canned ("no, isn't what we want", "is what we want, but doesn't actually give us more revenue", "too hard to sell", "we have different ideas now"). Half of the rest are shelved, to maybe tried later again. The rest is developed further ("we now also want an interface to manage it, instead of typing in database statements", "we want to control who gets access to it", "can we have it in green?"). Successful projects (and by definition, a project cannot be successful unless it has the operational data to back its success up) may be developed further. Oh, and what's a success now may no longer be a success 6 months from now. Because the market may have changed.

What I do know is that if I first spend a few months designing the ultimate way of doing something, then some time implementing, my customer cannot make any progress (testing it, demoing it to his customer, making revenue, finding out what the next step is) during those months, and, more importantly, I'm robbing the entire chain (me, my team, my customers, the company, the customers the company is serving) the opportunity of early failure.

sundialsvc4: First, please let me say that I pretty much always enjoy reading your various mediations, expositions, etc. Thank you very much - they are often thought-provoking and usually educational, and I appreciate the time and the effort you put into them.

In this case, though, I have to say that I disagree with you very strongly. This is going to be rather long and somewhat impassioned; please pardon. As my excuse, I offer the fact that I *often* have to go through the short-but-full-of-sound-and-fury version of this with clients, and have in fact just done so (last night.)

I worked on a Perl app for a couple of months, then wrote all the code in a week.

What made the "couple of months" before that week not scrap? What made the "several revisions" and the "discoveries" not scrap? Is it simply the fact that you weren't writing code when that was happening? Some people "think better on paper", or at the keyboard, at some level of the development process. I certainly don't need to experiment with, say, 'grep' or 'map' to know how they work, so I'm not going to do that. I don't have any doubts about writing a subroutine, or figuring out what it will return, etc., etc., etc... but there's *no* way that I can tell, without writing some code and putting it through the mill how my client, or the people that work for him, are going to react to and interact with an interface that I'm going to use as the front end to the app that I've been hired to write. There's *no* way for me to predict that, once I've followed all the specs that were written down and finished all the coding, the client's not going to look aghast and say "but of course you needed to make sure the numbers were all written out as words... in the Tarahumara language. In 144-point Comic Sans, in transluscent blue with a slight hint of iridescence, and displayed on multiple monitors - one per letter. It's standard in our industry!"

The one thing I've learned over the many years of dealing with clients is that Revisions Will Happen - at every single level of the project, including and especially those that you think you've nailed down and cast in stone. As a result, I make it explicitly clear (some might say "to a ridiculous degree") that it is the client's job and proper expectation to pay for those changes. Yes, including my learning curve - if that's what you want to call it. I almost don't care about the rest of the agreement, but that one part must be understood and signed off on.

That "learning curve", you see, is inevitable. It is impossible for me to know every single detail of every single type of business that I code for - and I am certainly not going to spend my (unpaid) time learning those details. That's really what this is about: who pays for that "wasted" time? To me, the answer is "the client". When they hire a new employee - say, an accountant - they inevitably have to retrain that accountant (and pay for that retraining time) to teach him the specifics of being an accountant in the ferret-shining industry rather than in the weasel-polishing trade that he came from. No one ever argues about this; it's a standard business expense.

In our business, however, you're supposed to intuit all of that data - because we programmers are so frightfully smart that, well, we should just be able to do that.

Um... HELL NO. I always gently-but-firmly shove that notion right back down the client's throat as soon as it comes out of one.

I was not “learning as I went.” First, I learned; then, I went.

Who paid for that learning time? If it was your client, you still, by your definition, "wasted" their money - no matter how you structure that explanation. If it was you... but then, I very much doubt that it was, or that it's even possible.

-- 
Education is not the filling of a pail, but the lighting of a fire.
 -- W. B. Yeats

This analogy is farcical.

Or perhaps not as it comes from the man who learnt everything he needs to know about software development from "Shop Class".

Can I recommend you a good surgeon that learnt his profession on butchery day in Home Economics.