I'd like to, at the very least have an alternative distribution of my program that doesn't have the CPAN modules installed, but has a install script that'll fetch the correct ones automatically and have these modules installed in the person's site or local perllib. Sounds exactly like what Module::Install is made for.

See if DistroBuilder is not the answer you're looking for.

Anyway, it's not a good idea to fetch fresh Perl modules before testing them with your application. Besides the best efforts, sometimes bugs are inserted with new features and the interface may be changed.