What is this based on? Pixies and fairy dust? ...
It's based on pragmatic experience. I've actually found it very useful for automating many tasks that happen alongside installation - hence my comments about using Module::Build for other things than installing modules.
However, I've never written an XS module, and I have no idea how well or otherwise Module::Build holds up when building external C code, but I meet regularly with some mongers who write XS code who swear by Module::Build. OK, if there are deficiencies here, please post feedback to the authors of the module. I don't feel it invalidates what I have to say in my review.