Until recently (see expat library issue) I've had success using modules from non-standard locations by adding to @INC the PERL5LIB environment variable and the LD_LIBRARY_PATH Module.

Admittedly my process is very manual intensive; building in the standard location then moving as required, but I have achieved the desired result