Maybe Test::Builder::Tester::Color would be of interest?

Worst case you probably could write a filter to postprocess your test output (i.e. something analogous to what colordiff does for diff output).