I'm the author of Dada Mail, the app used as an example.
I thought I'd give a quick, "how do I do" for bundling the CPAN modules into Dada Mail.
As markjugg pointed out, the audience for Dada Mail is more likely to be your casual website owner, small business person, etc - not a fellow developer. As such, these people may know how to sling some HTML to a very good degree and will know how to configure fairly complex web apps that have simple configurations, like Dada Mail, Moveable Type, etc but WILL NOT be able to easily jump into a shell and fire up CPAN. It's also one more step in setting up a program and with this audience, too many steps leads to your application being passed by for a different alternative. Some examples of similar applications that I have been told are too hard to either use or to install are Gnu Mailman and actually a php application, called PHPList. I haven't installed either myself.
I keep a record of the CPAN modules I need for Dada Mail in a CPAN Bundle - Here it is: http://cvs.sourceforge.net/viewcvs.py/*checkout*/mojomail/dada_mail_stable/dada/extras/developers/DadaMail.pm
It's in a CVS repository, so any changes, of course, will be annotated to a ridiculous degree. Any other security advisories about the program would come from the regular sources. These are usually few and far between.
Since the Bundle file is POD, it's also used to make documentation: http://mojo.skazat.com/support/documentation/DadaMail.pm.html
When I make a distribution of Dada Mail, I'll run cpan myself and get a fresh copy of all the needed modules and test with this setup. If something breaks, I'll know about it before it gets released. Just to clarify: the only thing in CVS is the Bundle File - no actual CPAN modules are kept in CPAN - always plucked fresh for each release (each release seems to be now about every 2 - 4 weeks)
As for, "What happens if there's a security problem". If there's a change in a CPAN module, the new version of Dada Mail will automatically have the fix and any testing done will see if anything breaks. (I don't have a test suite, but it's in the pipeline)
Fixing this type of problem basically means reinstalling the entire distribution of Dada Mail again. If you're a little slicker and know how to use cpan and have root access, you can basically just make a custom distribution of Dada Mail for your specific environment by removing the, "perllib" directory that comes with Dada Mail and running the DadaMail.pm bundle locally. This following scenario is mentioned in the POD of the Bundle file itself. I don't see this being any more dangerous than allowing a user to have their of person perllib in the home directory.
I think there's an added benifit of bundling your own CPAN modules with your application and that is: different version of CPAN modules do act differently, as markjugg pointed out. For example, Dada Mail relies on the HTML::FromText CPAN module, but can only use version 1.005 of the module and nothing later. This is because the 2.x version of the module needs a much larger CPAN module requirement itself, some of these modules being XS modules, which, as a rule, isn't something I attempt to require, since the target audience won't know/doesn't have the correct resources to compile code.
What markjugg is dreaming of may be actually easier to handle if it was the responsibilty of the app distributor (for the time being). I strung my current build tools with a simple shell script and a slightly custom CPAN config. I talk about how I did this here: CPAN Shell, Bundles and downloading specific versions and here: Install a CPAN module at x location
There has been an instance of Dada Mail using a CPAN module and customizing the CPAN module itself to make it actually work very different than before. This CPAN module is now not a prereq and isn't a part of the Bundle. I don't see this as a problem, as it's sort of the whole idea of Free Software: you put something out because you think it's useful, allowing people the freedom to use and modify it as they see fit.
Just quickly glancing at the Bundle file for Dada Mail, you'll notice that it relies on a fair amount of CPAN code - about 3.2 megs. This includes some big ones, like MIME-Tools. I would say the rest of the app is about .5 megs. Dada Mail would not be even close to what it is now without CPAN. Dada Mail is very far in its life as a program and any program of this complexity will have problems dealing with all the different pieces. Knowing exactly which CPAN modules are being used in each version greatly simplifies making sure the app you ship will work. I can bet you the popularity of the program would not be as great as it is now if the CPAN modules were not installed automatically, or, if I only used the core modules.
Hope that enlightens a bit.