If you're doing anything "serious" never run with the OS' perl install. That's just asking for trouble because at some point something is going to break that under you. Something will pull in an upgrade without you knowing and then you're going to be stuck trying to backtrack to working. Especially if you're not on your own machine and there's other admins in the mix.

Always compile your own application install of perl and use PATH and #!/usr/bin/env perl shebangs to find the one you want. What exactly that means is up to you and your mentioned options are both OK. I'd tried perlbrew, never really found it to be compelling enough versus installing to a fixed /somewhere/app/perl/X.YY.ZZ tree and using that and the environment. YMMV though and I'm sure you'll get more comments in its favor (maybe some against . . .).

Edit: Maybe to provide more clarification as to why I tend not to use perlbrew: I manage the perl install for a small-ish (tens of developers) group that manages the "glue" code piping everything together for a (fairly) large custom code base. We have production runs daily so stability of the platform and integrity of numerical results is one of our big "needs". We tend not to upgrade perl versions frequently because to do so would open up needing to do fairly extensive testing to make sure we're not (say) picking up unknown/unintentional numeric drift because something in a floating point library changed (which we've seen happen before). So there's not as much call to have a dynamic pick-and-choose from whatever perl version(s) capacity for our environment, and a fixed install every n years works out OK.

As far as modules, I've toyed with building in a custom symlink'd directory (theoretically pointing to a dated directory underneath that could be swapped) into the PERL5LIB baked into the binary and installled things there into the perl install. That's worked OK in general, but then since doing that haven't tried to do a full "upgrade" of the installed module set so there could be issues with that I never exercised. I'm planning on using local::lib to hold our "current" module set for the next full install we do so we'll see how that goes then . . .

Edit: and to answer your question not sure without seeing the entire exact error message but yes that sounds like you've got XS modules compiled for an older perl now not wanting to load into a newer version. Reinstalling things from CPAN will probably clean those up.