If you can live with the versions of the modules that come with the system's package management, I'd go with that — it's usually the easiest way.  If you need/want bleeding edge versions which in turn might require bleeding edge versions of other modules already installed with the system, there's the potential risk of destabilizing system tools that rely on the specific Perl that's originally shipped with the system, in case you simply install those newer versions over the existing ones... (installing your own stuff (with all its incompatible dependencies) into a separate directory is fine, though).

In such cases, I usually compile/install my own Perl under /usr/local, so I can mess with it at free will, without interfering with the system Perl. You'd then install modules via the cpan shell. And use lib ... isn't required in this case, because the perl binary under /usr/local already knows where to look for its modules...