Personally, I would look at CPAN::Mini or Pinto for maintaining an "approved" CPAN locally. The VM should only install from that approved CPAN.

You can take a look at the CPAN distroprefs for setting up your local compilation chain for problematic modules with non-CPAN extensions, but for those, I would consider building PPMs and install these. I don't know how Pinto handles non-CPAN binary blobs like PPM or other files.