Why do large projects have such an abysmal failure rate? Large companies that can afford to throw a half million or more at a project don't do so on a whim. They spec out the project and determine if it's feasible. And despite this, it still fails. Why?

It fails due to information costs. Those costs are frequently not considered when the project starts and the larger the project is, the greater the informatin costs. For example, Alice is the boss of Bill and Charlie. Alice happens to tell Bill something that Charlie should know and expects him to tell Charlie. Bill tells Charlie Bill's understanding of what Alice said and Charlie writes bad code because his specs are wrong. However, he doesn't know to ask Alice what the correct specs should be because he had no reason to think he was heading down the wrong path.

Obviously, if you have one manager and only two programmers, this is less likely to happen. However, as a project grows, these sorts of miscommunications continue to occur and the poor Charlies realize something is wrong so they keep trying to double-check their information and this raises overall costs (which is why I refer to "information costs", sometimes known as "search costs").

Part of the reason XP is so successful is that it defines a lightweight system where everyone has a clearcut role and you know exactly where you should turn for any information you need. Everything in successful project management is about lowering these search costs. Any time someone has a question, they can immediately receive timely, accurate and complete information. If you can pull that off, you'll probably be a successful project manager. Mind you, this isn't saying you should go the XP route, but if you do, it's frequently a quick route to success.