using

use lib [...]
[download]

is ok if all your scripts need to use that paths in all your environments, but that doesn't seem to be what you need.
A better way of controlling include paths that would allow you to manage the different requirements of different servers is to use local::lib. It may seem a bit complicated at first but it really isn't. Just follow the description of "The bootstrapping technique" to set up your local library path in the .bashrc (or csh/tcsh whatever you are using) script in the $HOME dir of each machine.
What that is doing is to set the PERL5LIB and some other environment variables for you so that a) @INC will include the correct path on each machine and b) modules will automatically be installed into the correct path using cpan(p|m).