As far as I understand it XP is mostly about eliminating project management by having the client sit down next to the programmer and telling him directly what he's expecting from the program.

I'm not saying that's a bad thing; most programmers could certainly do with more direct communication with the client. I do think it doesn't solve the problems in a project that's already running - the client simply never has the time to do this. Last project I did that went horribly wrong, we had more than 10 programmers working at the project 8 hours of every working day - we were lucky if the client could spare one person 4 hours per week to help us out with the specs. Ofcourse almost all of that time was spend by management promising the client golden castles in the sky instead. This particular project went over the deadline by more than 20 man-months (original estimate: slightly over two man-months - yes, the estimates were that bad)

And ofcourse, the client expects the project managers & programmers to be clairvoyant anyway :-).

I think most of the problems in the project described above were caused by just plain awful specs and cost-estimates coupled with horrible management from all sides.

I think the most important lesson I learned from that project was that you simply cannot spec & build anything without really understand what the client wants - not what they tell you they want: what they really want - what their goal is, how much they want to spend on it, and what their priorities are.

Let's back up a bit from there: I think I've spotted a recurring theme from all the projects I've done that went bad (as in: over budget and/or made me feel terrible and stressed out)

The most important thing you need from a client is a list of functionalities ordered by priority. Get this list before you start designing. It will make the client think about what it is they want, and you can (hopefully) see from that list what it is they actually think is important about the product. At least 50% of the time, you'll be really surprised (and then you can probably talk him out of paying for weeks of development on that really annoying piece of functionality that the client actually doesn't give a damn about, but was just put in the specs to stop somebody nagging in a meeting).

I quit that job by the way - I feel much better now that I'm self-employed :-)