If the extra directories have been compiled into your copy of perl itself then it's not so easy.
I think it's pretty easy - the perlbrew installs on my machine only have four entries in @INC, and building a copy of Perl without perlbrew is pretty easy too.
$ wget https://www.cpan.org/src/5.0/perl-5.32.0.tar.xz
$ tar -xaf perl-5.32.0.tar.xz
$ cd perl-5.32.0
$ sh Configure -de -Dprefix=/opt/perl5.32
$ make -j4
$ TEST_JOBS=4 make test_harness
$ make install
$ /opt/perl5.32/bin/perl -le 'print for @INC'
/opt/perl5.32/lib/site_perl/5.32.0/x86_64-linux
/opt/perl5.32/lib/site_perl/5.32.0
/opt/perl5.32/lib/5.32.0/x86_64-linux
/opt/perl5.32/lib/5.32.0