This is definitely along the lines of what I want to do.

I put the Config_m.pm file in ~/perl_libs, removed everything I could find related to cpanm, and changed my install line to be:

wget -O- http://cpanmin.us | /usr/bin/perl -I /home/xorl/perl_libs -MC
+onfig_m - App::cpanminus -f
[download]

Unfortunately it is still installing in the wrong place.

/usr/bin/perl -I /home/xorl/perl_libs -MConfig_m -MConfig -e 'print $Config{"installsitebin"} . "\n";' gives the "correct" value (/usr/bin). So I'm not sure what is going on at this point.