If you are using

set PERL_MB_OPT=--install_base /home/user/stuff
set PERL_MM_OPT=INSTALL_BASE=/home/user/stuff
[download]

then you need

set PERL5LIB=/home/user/stuff/lib/perl5
[download]