Good idea. Take a look at Module::Load. I've used it in some of my distributions where I've needed to dynamically load modules by module or file name based on conditions and whether they are available. May be of use to you here.

Regarding the sub redefined issue, you're not copying a module to a new name without changing the package name within for testing are you? I can't trigger this unless I copy one module, as-is, to another module name. eg: copy lib/Module.pm to lib/ModuleWithoutTests.pm without changing the package name within the copied file.

Granted though, I only spent a couple of minutes playing so I very well could be overlooking something...