You may want to check out Module::Install and its auto_bundle function.

fwiw, if you already have a directory with a bunch of dists unpacked inside it, a fairly blank Makefile.PL will automatically recurse. (ExtUtils::MakeMaker will recurse, but Module::Build won't.)