I already install cpan modules without cpan sudo (into my $HOME directory).

You're still messing with the system Perl by overriding which modules get loaded by Perl-based applications. Now, that risk isn't large, because very little actually uses the system Perl. But if you wanted to go the safe route, that would be to install your own Perl. This also have some major benefits: Control over the version of Perl, and the ability to upgrade Perl without breaking anything. As a big-time Perl user, these features should be appealing to you.