For example with cpanp its by default

cpanplus\01mailrc.txt.gz
cpanplus\02packages.details.txt.gz
cpanplus\03modlist.data.gz
cpanplus\db.sql
cpanplus\5.16.1\build\ack-2.08\t\swamp\Makefile.PL
cpanplus\authors\id\R\RU\RURBAN\CHECKSUMS
cpanplus\authors\id\R\RU\RURBAN\Compress-Bzip2-2.17.tar.gz
cpanplus\install-logs\ack-1.96-1371720568.log
[download]

the sources are shared, the build directories are not

You can change location from "$HOME" with PERL5_CPANPLUS_HOME

With cpanm its similar , to keep it from downloading source files over and over again (default) just tell it to cache them

export PERL_CPANM_OPT="--cascade-search --save-dists=$HOME/.cpanm/cach
+e --mirror=$HOME/.cpanm/cache --mirror=http://search.cpan.org/CPAN"
[download]

With cpan its also very similar, $HOME by default ... can use https://metacpan.org/pod/cpan#CPAN_OPTS to use -j alternate/configfile ... here is what installing CGI using 3 different perls looks like

.cpan\FTPstats.yml
.cpan\histfile
.cpan\Metadata
.cpan\build\CGI-4.21-0lIz7E.yml
.cpan\build\CGI-4.21-f0xo4H.yml
.cpan\build\CGI-4.21-v1xWJv.yml
.cpan\build\CGI-4.21-0lIz7E\Changes
.cpan\build\CGI-4.21-f0xo4H\Changes
.cpan\build\CGI-4.21-v1xWJv\Changes
.cpan\CPAN\MyConfig.pm
.cpan\sources\MIRRORED.BY
.cpan\sources\authors\01mailrc.txt.gz
.cpan\sources\authors\id\L\LE\LEEJO\CGI-4.21.tar.gz
.cpan\sources\authors\id\L\LE\LEEJO\CHECKSUMS
.cpan\sources\modules\02packages.details.txt.gz
.cpan\sources\modules\03modlist.data.gz
[download]

You don't need perlbrew for any of this, and its not exactly a secret

I don't use cpanp or cpan anymore, I mostly use cpanm with the default settings (no caching)