Well, it seems like there are several ways to go. Here's one. Create a module that exports plan() and ok() by itself (i.e. fakes them without using Test::More). Then distribute that module along with the test and put the code below at the top of the test. This way you only write your tests once, rather than two versions of all tests like you have.
BEGIN {
eval {
use Test::More;
};
if ($@) {
use myModuleThatBehavesLikeTestMore;
}
}