Given that I tend to ignore these things for both apps and modules, I don't really follow the benefit. But, if others like it, that's cool.
The biggest problem I see is that CPAN is built for installing modules. There's a lot more to installing an application. Take a look at how Trac installs for what I consider to be the bare minimum. Applications, frankly, should be installed in a single click. You don't do that from CPAN.
Maybe someone should write a ports interface to CPAN.
My criteria for good software:
- Does it work?
- Can someone else come in, make a change, and be reasonably certain no bugs were introduced?