You don't say so, but if the machines exist on the same network, I would be inclined (after testing of course) to share out the perl library directories from a central server. This allows you to maintain one directory tree, instead of having to physically/logically touch each machine.

Also, you don't mention whether or not you use the same perl version across all of the systems. If not, you will wind up testing each and every combination.

If neither is an option, I would set up a multi-boot system with all (4?) systems available. Then, set up a master directory structure that includes the sub directories for each architecture and perl version.

Super Search will give you pointers to how to set up a "private" perl library structure. Once that structure is set up, you will be able to simply install the appropriate modules using a combination of CPAN (or CPANPLUS) and the individual environments. Simply build (and test) for one environment (say, 5.4 x86_64) then create an autobundle module. Now, use that to create the other environments, then install the entire environment on a shared drive (if possible) or rsync the entire directory structure on each machine.