You seem to be thinking in terms of many independent applications that exist seperately and are started by huge copy and pastes

Not at all. I am, however, defining an "application" as being one customer demand, which may be different than how you are thinking of it. So if I have a CRUD app, this will be:

• 1 CGI App module
• some number of backend modules
• some number of templates
• 1 instance script

When someone asks for a second CRUD "app", I really just create a new instance, but I might edit the App module to allow for new features. So:

• The original App Module is edited (not a copy, just updated)
• some new templates are added. Any templates that aren't overridden are inherited.
• a new instance script

So are you saying that to create this second app, the developer checks out the ORIGINAL app, and adds this second instance to it? That would solve all the updating issues, bu where do you draw the line? Is there a line? The developer is checking out ALL code? If I put the instance scripts and templates in the same tag as the app module, I can't install the app module on another machine without installing the instance, which is not desired.

Okay, we've looked at that model. The problem was trying to keep development from clogging up production. Quite often we'll start on development and get pulled off onto other projects, leaving code incomplete, sometimes returned to, sometimes never returned to. How do we merge the development and production trees cleanly? Won't the development tree get all sorts of hanging code? (which is a current problem for us)

(eg tagging the release and running your test suite)

Here I'm missing details again. If we're checking out the entire codebase, are we running every bit of test code we have? How do we do the install process so that only the changed bits of code are tested on the new machine?

Believe me, I think we're talking about the same desires, I've just failed to get a practical, working system every time I've tried, and I think it's because I'm doing too much guessing on how to implement these things.

We've come up with systems that would work great in 15 person shops, but in my 1-3 person shop, with a constant backlog of tasks, rapidly shifting priorities, a windows-based designer (vs the Linux-based coders and servers), I've been unable to implement a setup that actually works. (Anyone with advice on what to do with the Windows designer who edits our templates would be welcome, since he can't test his edits on his machine.)

Script how to install version X of Foo on machine Y.

This makes it sound as if you AREN'T checking out the entire code base, so I'm confused again. An "app" could be one codebase, but we have modules that inherit from others, and template sets that inherit from others, so I'm not seeing where to draw the line between apps. And of course, we have support modules, with their own dependancies.

part of your install process

What is your install process? Since you're talking about running tests and install process, it sounds like it's a bit more than a check out. What do you do?

A good strategy is to take half your machines out of the load balancer, install there, tell the load balancer to switch which machines are online, install on the rest, then bring the rest back online. That way at all points all webservers are consistent.

I only wish I had the budget for load balancing. I've been trying to get a test server for years. We have a number of servers for different audiences for security reasons, and a generally common codebase between them.

Really, I'm not disagreeing with good practices, I'm just trying to figure out how to IMPLEMENT them.