I use perlbrew for Perl, and I use Homebrew to install external libraries. The key thing I learnt when setting this up was to modify my .profile file - it currently looks like this:

source ~/perl5/perlbrew/etc/bashrc
export PATH="/usr/local/opt/libxml2/bin:$PATH"
export PATH=/usr/local/bin:/usr/local/sbin:~/bin:$PATH
[download]

The first line means that when I fire up perl in the terminal it will use the current perlbrew version of Perl, which also means that cpan will install modules in the correct location for that Perl.

The second line I think is a hangover from MacPorts and you probably don't need it.

The third line ensures that Perl can find external libraries installed by Homebrew - I have seen different configurations posted but this one has always worked for me.