Thank you all for the very helpful replies/comments on this. I have learned a LOT from them. To summarize, there seem to be 4 basic approaches that are on a scale that ranges from on the one hand running straight from the source files and on the other hand having everything packaged up in CPAN-like modules distributions. There are additional points in between these 4 on the scale.

• Run everything from library module sources and not bundle anything into a module (unless you have to for XS support or something similar). This is described by perrin here, here, here, and here. This is basically what I do currently except many of the modules are lacking unit tests and sufficient pod documentation on their interface. ExtUtils::MakeMaker would not be needed dat all.
• Make a single module distribution and stick everything in ./lib. It would support either having or not having a build process (Makefile.PL), with tradeoffs, depending on whether the application pointed to ./lib directly or not. This is actually very similar to the previous bullet item, but does let you "borrow" some ExtUtils::MakeMaker functionality. If a library needed to be broken up into module distributions for CPAN or some other reason, I think this would be a good first step.
• Make a single module distribution but with submodules (for lack of a better name) that sit in subdirectories and each have their own Makefile.PL and their own unit tests. The top level module sits above them with its own Makefile.PL and integration tests. How to set this up is described by adrianh here. This is a useful technique that I did not know existed. I think this would be a good second step in moving towards individual module distributions, and it seems to pretty much make the the global makefile for you.
• Bundle everything up into various CPAN-like modules. Unless a particular module distribution really needed to be separate from the rest (to release to CPAN or for some other reason), it would probably not make much sense to go to this step.