Assuming you have lib and lib/perl5 in your home directory, have you tried the full path rather than using '~'?

perl Makefile.PL PREFIX=/home/XXX/lib/perl5

At times, I've run into problems using '~' to specify a home directory.