Thanks Prove looks very close to what I want,
I played around a bit with Build.PL and setting up custom ACTION_test subclasses, but that seemed very clunky (they do say its alpha) and difficult to maintain.