My guess is that all modules using Module::Build didn't properly install, as older versions of Module::Build don't respect PREFIX. But installing Perl locally has been a longstanding practice and I recommend it over using the system-installed Perl in any situation, because it gives you more control and sanity.