I could do the build in /opt/ I do that now by placing perl in /opt/perl.
I'm looking at moving to ARM. The issue I see is that building CPAN modules is not possible via cross-compiling.
Compiling under ARM for ARM sucks because it is just not fast enough. For that I think I may have to deploy a x86 dev box and ARM dev box. cross-compile on the PC then ssh over to the ARM machine for the CPAN module build segment.