According to the perlbrew site, there is an option available for installing a cpanm instance that is shared across the different perlbrew perls.

perlbrew install-cpanm
[download]

I think it will store it's configuration and data files (such as sources) in ~/.cpanm. I do not see similar support for cpan. I have not used this feature of perlbrew, so if I'm not correct, I hope someone will correct me.