There's nothing preventing cpanm from being used on private repositories. Recently Pinto was announced right here at the Monastery: Announcing: Pinto. It seems to be designed to assist with at least some of the issues you mentioned (Speed isn't addressed. Run your tests in parallel, and re-run sequentially those that fail.)

cpanm is a pretty powerful tool though; you can point it at a link to a tarball of a distribution, for example. You could literally maintain a site that contains all the distributions needed for your code, and then use Perl and cpanm to script the full install.