Why write separate test files if you're going to run them together? Alternately posed, why mash test files together under one runner if you've already separated them?

In place of segments, I'd use subdirectories under t/. Then you can use prove to run individual segments. (If you're using Module::Build or even ExtUtils::MakeMaker, you can add your own ./Build or make target to run the tests for individual segments.

Your make test (or ./Build test) should run all tests under t/ effectively, though (I forget if) you may have to specify recursive traversal of t/.