If you use the perl executable provided by your system vendor, only install modules into it through your package manager, also provided by your system vendor.

If you use your perl, compiled by yourself, only install modules into it using cpan.

Leave the two separate, and in my opinion, leave the system Perl to the system vendor, because your whole system will break in various parts if you install/upgrade to an incompatible version of a module that is crucial, for example for your system package manager, if that happens to be written in Perl.