No, I am not saying that I think that they made a good design decision. Rather, I am saying that I can think of many additional further mistakes which they could make which could result in the same symptoms, but without which their performance might have been still OK. Here is a sample:

• Is there a good one-to-one mapping between database-backed objects and application level screens? If there is then you have surprisingly little need for joins. You can't always aim for this, of course. But when you get it in a web-based environment, you literally can't make much use of joins.
• Did they have appropriate indexes? Indexes aren't only used for joins. If you ask for all accounts that are represented by representative X, having an index on account_rep_id will be useful whether or not you are joining.
• Did they use placeholders or an equivalent? From your description we can answer that in the negative since the WHERE clause was passed in as a string. I believe this to be a worse decision than the lack of joins. First there are the security reasons stated in Use placeholders. For SECURITY!. But secondly some databases (Oracle would be a good example) go out of their way to try to cache used queries, and contention for that cache will make them fall over fast if you force them to parse everything. (Oracle specifically has a setting that makes it rewrite everything with placeholders. This doesn't solve the security issue, but might help their performance issues.)
• Were they using Enterprise Java beans? Multiple Java experts of my aquaintance have told me that performance with beans ranges from bad to worse. I have heard of plenty of projects with good performance using direct DAO. I've heard of none with beans.

UPDATE:

The points you make might show that the performance isn't due to this design decision, but I don't think it solves the points about maintainability. In particular, the OP noted that the programmers didn't have an understanding of the user requirements. No ammount of good design will save you if you don't really understand the problem.

----
I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
-- Schemer

: () { :|:& };:

Note: All code is untested, unless otherwise stated

I wasn't saying that this design decision didn't cause the performance problem. I was instead saying that without a detailed post-mortem, you shouldn't conclude that it was that. Personally I'm never comfortable with conclusions unless I have been balancing at least three theories. One theory is usually prejudice. 2 gets me into binary thinking where I have a hard time really thinking through the merits of either. 3 seems to be my threshold to really start breaking things down and identifying what is going on.

Also when a project goes wrong, it usually goes way wrong. My suspicion is that the database decision is more of a symptom than a cause. The poor understanding of user requirements is another symptom. Firing the project manager is suspicious. I'm sure that if I knew more about the project failure, I would come up with lots more symptoms.

Symptoms of what, though, is a different story. Were key people not properly prepared for their roles? Does the company have a command and control philosophy which results in more CYA than useful early feedback about potential problems? Was the wrong kind of project structure for the problem in use? Did you have a deathmarch? (These are not mutually exclusive possibilities...)

There are plenty of classic books from Peopleware which have lots of useful stuff to say on why projects fail and what you can do about it. But without knowing some details, there isn't much that you can concretely say about what should have been done better on any given project.

Which is why I limited myself to pointing out other possible technical performance problems other than the obvious which plausibly played a role.

Just a nitpick - you probably mean Enterprise Java Beans instead of Java Beans. The former are the Java-blessed way of creating distributable components, the latter a fairly simple specification for exposing properties of an object. It's unfortunate they're so similarly named.

Chris
M-x auto-bs-mode

Abigail

I feel you're judging the programmers a bit harshly. Unfortunately, even if one makes one's objections known in discussions (if there are any to start with, this depends on company culture), there not always taken into account.

You start from the assumption that managers and senior designers/programmers will listen to reason and follow common sense arguments, which unfortunately is not realistic in all cases.

Personally I've been lucky in this respect, and, judging from your post, so have you, but I'm afraid it's a mistake to take it for granted.

Just my 2 cents, -gjb-

Update: Well, being nearly fired counts as being lucky in my book.

Although I've never met Abigail, I'm pretty sure I speak for him (and most others) when I say we have not been similarly lucky. I remember nearly being fired when pointing out the deficiencies in a given design. (And, I'm sure it was part of the reason why one of my contracts was ended early.)

And, there's no reason why you have to follow the designs given, if they're stupid. If I was a programmer on that project, I would implement my own DAOs and write it that way. Then, when my one piece has good performance with the ability to choose which DAO to use in a given situation without impact ... results speak for themselves much louder than anything else ever can. (I've done this method, too.)

------
We are the carpenters and bricklayers of the Information Age.

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

It all depends on how the company defines the responsibility of each role. It would be unfair to sack a person, if one’s role even does not allow/give the opportunity to question.

Theoretically, if some one with a higher rank approved your coding or design, you now holds much less responsibility on your own, instead the approvers are the main focus. If approval is just a signature with no responsibility and risk, it is too easy to be a manager, and the organization will not roll.

I agree with you that the designer should be sacked/demoted, as he is not capable. In this case, programmers are innocent, first they were not given opportunity to question, and secondly I don’t think the company expected them to have enough knowledge to question. (I am not saying that they really didn’t have the knowledge, what I am saying is that one should not be blamed for not delivering, if they were not responsible for delivering.)

Theoretically, if some one with a higher rank approved your coding or design, you now holds much less responsibility on your own, instead the approvers are the main focus. If approval is just a signature with no responsibility and risk, it is too easy to be a manager, and the organization will not roll.

I've worked both in US and European "corporate cultures", and I've often been asked about differences. This is one of the differences I often point out: the American tendency to avoid responsibility, and to look for a scape goat. As long as I've a signature of some manager, I'm safe. Or Yes, I know it's foolish, but he's senior management, I'm not going to question him. I think this is also the reason why in the USA there's less initiative coming from the 'lower ranks' that it happens in Europe. Perhaps that's why workers in Europe have the same productivity as US ones, while working far less hours.

Abigail

... I guess said programmer never really understood that the R stands for relational in RDBMS nor why X normal form is better than X-1 normal form.

This is the most asinine thing I've ever heard of. Thank you for sharing it. I now feel as if a hole in my being has been filled.

I may be a little ironical here, but I believe the problem with RDBMS is that if you got a RDBMS everything looks like a table. There are many ways to store information around, but it seems the only "reasonable" way to do it is to hack together some sort of normalized form. This may be OK in some situations, but not in each and every one.

Very often I believe a file, often a text one, might be more than enough: imagine if you had to store complex tree-like structures like a parsed programming language of your choice in a RDBMS!

Then you are free to create second-level databases of your choice; but when you start to have tens of "link tables" just to trace trees or N:N relationships....

From the point of view of pure theory, the designer was still under the relational system. It's just that the relational bit was being done at the application layer instead of the database.

----
I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
-- Schemer

: () { :|:& };:

Note: All code is untested, unless otherwise stated

If the target database supports it, I place as much of the sql as possible in a stored proc layer. The stored proc layer can/should contain procs for retrieving, inserting, modifying and deleting business objects - let the stored proc layer handle all the joining and other table maintenance. You as a front-end developer need not worry about it. Once that stored proc layer is in place, you can then automatically create a thin perl wrapper around the stored procs (query the db, find the procs and params - write code to create code). You may need to manually tweak the module but if the proc layer is competently designed and implemented, you shouldn't need to. In the past, I've coded up the thin layer as a 'uses' module that just basically passes everything off to a contained DBI object. You could always subclass DBI, but I always have a more comfortable level by constraining DBI than opening it up for full use.

-derby

Yep. As derby proposes ... write the SQL as optimal as you can, wrap it up as stored procs and then generate the DAO. In the project I spend most of my time on we have some 700 stored procs that need to be accessed from ASP and VB code ... what could be easier than to whip up some Perl code that'll connect to the database, load the info about the stored procs and generate the whole data access layer.

Sure the layer will not be any dandy OO, but if you wanted that you could put another layer on top of this thing very easily. I never felt the need.

Jenda
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
   -- Rick Osborne

Edit by castaway: Closed small tag in signature

Jus to stay on the off-topic side of life, we also had a number of memory leaks found, because the whole project was designed by some Smalltalk "expert" and the implemented in Visual Basic (!), and we kept on discovering weird pointers problems and memory leaks all around.

When goal displacements like this happen, it's often instructive to look at them from the perspective of incentives, and look for reasons why the designer favored reuse over use. Sometimes it'll be for reasons deep within the designers head that you can't hope to get at, sometimes it'll be for reasons that you can get through careful (and tactful) questioning, but sometimes the reasons are non-obviously external.

The times I've seen designers (and teams) go wonky over reuse, it's usually been because upper management's fad-of-the-week became reuse, and management either offered incentives (e.g., "part of your bonus will be based on reuse") or they pounded the table demanding reuse "or heads will roll". Either way, common sense started a slow sideways move toward the door.

My comiserations for your loss pg, you say they've fired people. That is fine and all , and happens - but did the remainder of the implementation team do any postmortem on why it all went worng. Too often I can recall dodging a bullet and just rushing headlong into the next diaster.

How about maintaining this shiny-new-sharp-edges-still-intact system?

The One Big Table design is hardly unique, but having a presumably reasonable database model and then ignoring its benifits (as the OP described) is something I've not heard of often.

----
I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
-- Schemer

: () { :|:& };:

Note: All code is untested, unless otherwise stated

The part of this story that strikes me as sadly negligent/incompetent is that the designer or manager either didn't bother creating a (good) proof of concept or prototype for what was clearly a fairly radical idea.

Experimenting is a good and positive thing, but only the silliest developer/manager would put experimental ideas into production w/o testing them first, or at least making sure that that more conventional alternatives could be quickly put in place.

Reuse is a good thing, too. Reuse of code via the use of proven APIs, CPAN modules, libraries, etc. is a truly wonderful thing. Stupid/forced/kludgy reuse (remisuse?) is inefficient at its best, and pernicious at its worst.