Your lib directory can be shared across architectures -- perl takes care of architecture-specific libraries quite nicely. Your only issue, then, is getting to the correct binary. I recommend something on the order of:

#!/bin/ksh
# If ksh isn't available for all platforms, use /bin/sh, 
#  but test the the argument quoting below -- not 
#   all /bin/sh are created equal...

# have this be file ~/bin/perl

ARCH=`uname`
exec ~/bin/perl-$ARCH "$@"
[download]

That is, place architecture-dependend binaries for perl in your ~/bin directory, and have ~/bin/perl redirect to the correct perl.
As I said above, they can all share ~/perl/lib (or whatever) as their install area for libraries.
I hope this helps...

--JAS