This is a real life story, and it actually got my immediate manager fired (of course this is only part of the reason why she got fired.)

We had a project to replace our entire merchandising system, which is currently running on IBM mainframe. According to the original project plan, it is supposed to be in production already, but it is not, and will not until next Sep. Luckily I was not part of the team. The project is not in Perl, so this is OT. But the lesson it taught is across languages, so I think just share.

Although the system will only be used by internal users, the team decided to make it a web application. That's cool, and I guess that's one of trend going on today.

The code is OO, and as a general principle, the team decided that all the database queries should be wrapped in DAO's (Data Access Object). So to the rest of the system, no code accesses database directly, instead, only accesses database through DAO's. This is still cool, and is still on the right track.

Then the next idea brought the whole design to a level of insanity I had never heard of before, actually even never ever had the slightest thought of. Unfortunately because it was proposed by a senior designer, and backed up by the manager, it got passed the design review, and even had everything implemented. The reusability of those DAO classes seemed to be so important to that designer, and through his past experience, he realized that all the SQL queries were different, thus the DAO's used to wrap them were not reusable. In order to have a quantum leap of reusability, he made a design rule to his team: there shall be no joins in SQL queries, and the join shall be handled in (programmer's own) code. Now instead of having a big number of DAO's for all SQL queries, they only needed to create one DAO for each table to wrap a single dynamic query. The query accepts a string parameter that will be used as where clause, and returns a result set that holds all columns of the table. Now the reusability is largely increased, as there would be only one DAO for each table, and it is reused whenever you need data from that table.

What a brilliant idea! What a stupid crime committed under the beautiful cover of reusability.

It was until the point they started to system test what they created, the team suddenly discovered a nightmare that could never be waken up:

Everyone likes reusability, and it is right and beautiful, but just like any other wonderful things, if you over do it, you kill all of its good purposes, and create evil.

You don't believe this story? me either, it just sounds so unreal, and I really wish it had never happened.

Replies are listed 'Best First'.
Re: The crime under reusability
by tilly (Archbishop) on Dec 01, 2003 at 15:14 UTC
    I'm going to go out on a limb and ask if the lack of joins is really the reason for the performance problems.

    No, I am not saying that I think that they made a good design decision. Rather, I am saying that I can think of many additional further mistakes which they could make which could result in the same symptoms, but without which their performance might have been still OK. Here is a sample:

    • Is there a good one-to-one mapping between database-backed objects and application level screens? If there is then you have surprisingly little need for joins. You can't always aim for this, of course. But when you get it in a web-based environment, you literally can't make much use of joins.
    • Did they have appropriate indexes? Indexes aren't only used for joins. If you ask for all accounts that are represented by representative X, having an index on account_rep_id will be useful whether or not you are joining.
    • Did they use placeholders or an equivalent? From your description we can answer that in the negative since the WHERE clause was passed in as a string. I believe this to be a worse decision than the lack of joins. First there are the security reasons stated in Use placeholders. For SECURITY!. But secondly some databases (Oracle would be a good example) go out of their way to try to cache used queries, and contention for that cache will make them fall over fast if you force them to parse everything. (Oracle specifically has a setting that makes it rewrite everything with placeholders. This doesn't solve the security issue, but might help their performance issues.)
    • Were they using Enterprise Java beans? Multiple Java experts of my aquaintance have told me that performance with beans ranges from bad to worse. I have heard of plenty of projects with good performance using direct DAO. I've heard of none with beans.
    And my point is that just because you know that they did one thing wrong, and had predictable problems, doesn't actually prove that the mistake you see is what actually went wrong.


    lachoy is exactly right. Updated.

      The points you make might show that the performance isn't due to this design decision, but I don't think it solves the points about maintainability. In particular, the OP noted that the programmers didn't have an understanding of the user requirements. No ammount of good design will save you if you don't really understand the problem.

      I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
      -- Schemer

      : () { :|:& };:

      Note: All code is untested, unless otherwise stated

        I wasn't saying that this design decision didn't cause the performance problem. I was instead saying that without a detailed post-mortem, you shouldn't conclude that it was that. Personally I'm never comfortable with conclusions unless I have been balancing at least three theories. One theory is usually prejudice. 2 gets me into binary thinking where I have a hard time really thinking through the merits of either. 3 seems to be my threshold to really start breaking things down and identifying what is going on.

        Also when a project goes wrong, it usually goes way wrong. My suspicion is that the database decision is more of a symptom than a cause. The poor understanding of user requirements is another symptom. Firing the project manager is suspicious. I'm sure that if I knew more about the project failure, I would come up with lots more symptoms.

        Symptoms of what, though, is a different story. Were key people not properly prepared for their roles? Does the company have a command and control philosophy which results in more CYA than useful early feedback about potential problems? Was the wrong kind of project structure for the problem in use? Did you have a deathmarch? (These are not mutually exclusive possibilities...)

        There are plenty of classic books from Peopleware which have lots of useful stuff to say on why projects fail and what you can do about it. But without knowing some details, there isn't much that you can concretely say about what should have been done better on any given project.

        Which is why I limited myself to pointing out other possible technical performance problems other than the obvious which plausibly played a role.

      Just a nitpick - you probably mean Enterprise Java Beans instead of Java Beans. The former are the Java-blessed way of creating distributable components, the latter a fairly simple specification for exposing properties of an object. It's unfortunate they're so similarly named.

      M-x auto-bs-mode

Re: The crime under reusability
by Abigail-II (Bishop) on Dec 01, 2003 at 10:17 UTC
    They only sacked the manager? What did they do with the senior designer? And what did they do with the programmers? I think the programmers are as much to blame as the designer; from your story it seems that they are mindless sheep, just implementing whatever gets passed to them from the higher ranks, without taking responsibility.


      I feel you're judging the programmers a bit harshly. Unfortunately, even if one makes one's objections known in discussions (if there are any to start with, this depends on company culture), there not always taken into account.

      You start from the assumption that managers and senior designers/programmers will listen to reason and follow common sense arguments, which unfortunately is not realistic in all cases.

      Personally I've been lucky in this respect, and, judging from your post, so have you, but I'm afraid it's a mistake to take it for granted.

      Just my 2 cents, -gjb-

      Update: Well, being nearly fired counts as being lucky in my book.

        Although I've never met Abigail, I'm pretty sure I speak for him (and most others) when I say we have not been similarly lucky. I remember nearly being fired when pointing out the deficiencies in a given design. (And, I'm sure it was part of the reason why one of my contracts was ended early.)

        And, there's no reason why you have to follow the designs given, if they're stupid. If I was a programmer on that project, I would implement my own DAOs and write it that way. Then, when my one piece has good performance with the ability to choose which DAO to use in a given situation without impact ... results speak for themselves much louder than anything else ever can. (I've done this method, too.)

        We are the carpenters and bricklayers of the Information Age.

        Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

      It all depends on how the company defines the responsibility of each role. It would be unfair to sack a person, if oneís role even does not allow/give the opportunity to question.

      Theoretically, if some one with a higher rank approved your coding or design, you now holds much less responsibility on your own, instead the approvers are the main focus. If approval is just a signature with no responsibility and risk, it is too easy to be a manager, and the organization will not roll.

      I agree with you that the designer should be sacked/demoted, as he is not capable. In this case, programmers are innocent, first they were not given opportunity to question, and secondly I donít think the company expected them to have enough knowledge to question. (I am not saying that they really didnít have the knowledge, what I am saying is that one should not be blamed for not delivering, if they were not responsible for delivering.)

        Theoretically, if some one with a higher rank approved your coding or design, you now holds much less responsibility on your own, instead the approvers are the main focus. If approval is just a signature with no responsibility and risk, it is too easy to be a manager, and the organization will not roll.

        I've worked both in US and European "corporate cultures", and I've often been asked about differences. This is one of the differences I often point out: the American tendency to avoid responsibility, and to look for a scape goat. As long as I've a signature of some manager, I'm safe. Or Yes, I know it's foolish, but he's senior management, I'm not going to question him. I think this is also the reason why in the USA there's less initiative coming from the 'lower ranks' that it happens in Europe. Perhaps that's why workers in Europe have the same productivity as US ones, while working far less hours.


Re: The crime under reusability
by !1 (Hermit) on Dec 01, 2003 at 07:49 UTC

    ... I guess said programmer never really understood that the R stands for relational in RDBMS nor why X normal form is better than X-1 normal form.

    This is the most asinine thing I've ever heard of. Thank you for sharing it. I now feel as if a hole in my being has been filled.

      I may be a little ironical here, but I believe the problem with RDBMS is that if you got a RDBMS everything looks like a table. There are many ways to store information around, but it seems the only "reasonable" way to do it is to hack together some sort of normalized form. This may be OK in some situations, but not in each and every one.

      Very often I believe a file, often a text one, might be more than enough: imagine if you had to store complex tree-like structures like a parsed programming language of your choice in a RDBMS!

      Then you are free to create second-level databases of your choice; but when you start to have tens of "link tables" just to trace trees or N:N relationships....

      From the point of view of pure theory, the designer was still under the relational system. It's just that the relational bit was being done at the application layer instead of the database.

      I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
      -- Schemer

      : () { :|:& };:

      Note: All code is untested, unless otherwise stated

Re: The crime under reusability
by BUU (Prior) on Dec 01, 2003 at 07:41 UTC
    Just to try to take this whole thread some place positive instead of just sitting around pointing and laughing at the manager, how would you have done it instead? Assuming you have to stick to the whole 'DAO' idea and abstract the sql away completely. Three hundred daos for each sql query? Something else?
      If the target database supports it, I place as much of the sql as possible in a stored proc layer. The stored proc layer can/should contain procs for retrieving, inserting, modifying and deleting business objects - let the stored proc layer handle all the joining and other table maintenance. You as a front-end developer need not worry about it. Once that stored proc layer is in place, you can then automatically create a thin perl wrapper around the stored procs (query the db, find the procs and params - write code to create code). You may need to manually tweak the module but if the proc layer is competently designed and implemented, you shouldn't need to. In the past, I've coded up the thin layer as a 'uses' module that just basically passes everything off to a contained DBI object. You could always subclass DBI, but I always have a more comfortable level by constraining DBI than opening it up for full use.


      Yep. As derby proposes ... write the SQL as optimal as you can, wrap it up as stored procs and then generate the DAO. In the project I spend most of my time on we have some 700 stored procs that need to be accessed from ASP and VB code ... what could be easier than to whip up some Perl code that'll connect to the database, load the info about the stored procs and generate the whole data access layer.

      Sure the layer will not be any dandy OO, but if you wanted that you could put another layer on top of this thing very easily. I never felt the need.

      Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
         -- Rick Osborne

      Edit by castaway: Closed small tag in signature

I work in one such shop
by l3nz (Friar) on Dec 01, 2003 at 11:54 UTC
    We have a large system which database is based on DB2 on OS/390. What we do is, when the application starts, we call a CICS module that sweeps the 100+ table database and returns a "recordset" of relevant information for each table that is then processed by the user level using ADO/DAO filters. This is rather clumsy, but it's by far more efficient that querying the database for each single program need (we do many million application runs a day). What's awful with it is the poor design of the application level, where there is no central repository for common useful routines and each programmer in a developer team of 30 to 50 people re-invented the wheel.... often with a variable number of sides :-)

    Jus to stay on the off-topic side of life, we also had a number of memory leaks found, because the whole project was designed by some Smalltalk "expert" and the implemented in Visual Basic (!), and we kept on discovering weird pointers problems and memory leaks all around.

Re: The crime under reusability
by dws (Chancellor) on Dec 01, 2003 at 17:07 UTC
    The reusability of those DAO classes seemed to be so important to that designer, and through his past experience, ...

    When goal displacements like this happen, it's often instructive to look at them from the perspective of incentives, and look for reasons why the designer favored reuse over use. Sometimes it'll be for reasons deep within the designers head that you can't hope to get at, sometimes it'll be for reasons that you can get through careful (and tactful) questioning, but sometimes the reasons are non-obviously external.

    The times I've seen designers (and teams) go wonky over reuse, it's usually been because upper management's fad-of-the-week became reuse, and management either offered incentives (e.g., "part of your bonus will be based on reuse") or they pounded the table demanding reuse "or heads will roll". Either way, common sense started a slow sideways move toward the door.

Re: The crime under reusability
by submersible_toaster (Chaplain) on Dec 01, 2003 at 12:26 UTC

    My comiserations for your loss pg, you say they've fired people. That is fine and all , and happens - but did the remainder of the implementation team do any postmortem on why it all went worng. Too often I can recall dodging a bullet and just rushing headlong into the next diaster.

    How about maintaining this shiny-new-sharp-edges-still-intact system?

    I can't believe it's not psellchecked
Re: The crime under reusability
by zby (Vicar) on Dec 01, 2003 at 10:22 UTC
    As allready mentioned this is not that unique in the corporate world. What do you think about fitting all data into one table, with columns field name and value, under the pursuit of simplicity of database access? That was the design of the main internal database in one of the polish biggest portals.

      The One Big Table design is hardly unique, but having a presumably reasonable database model and then ignoring its benifits (as the OP described) is something I've not heard of often.

      I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
      -- Schemer

      : () { :|:& };:

      Note: All code is untested, unless otherwise stated

Re: The crime under reusability
by Roger (Parson) on Dec 01, 2003 at 06:29 UTC
    It's all company politics, welcome to the real world. ;-)

Re: The crime under reusability
by rdfield (Priest) on Dec 01, 2003 at 12:58 UTC
Re: The crime under reusability
by Art_XIV (Hermit) on Dec 02, 2003 at 16:00 UTC

    The part of this story that strikes me as sadly negligent/incompetent is that the designer or manager either didn't bother creating a (good) proof of concept or prototype for what was clearly a fairly radical idea.

    Experimenting is a good and positive thing, but only the silliest developer/manager would put experimental ideas into production w/o testing them first, or at least making sure that that more conventional alternatives could be quickly put in place.

    Reuse is a good thing, too. Reuse of code via the use of proven APIs, CPAN modules, libraries, etc. is a truly wonderful thing. Stupid/forced/kludgy reuse (remisuse?) is inefficient at its best, and pernicious at its worst.

    Hanlon's Razor - "Never attribute to malice that which can be adequately explained by stupidity"