Personally, I would prefer if Prima were smarter about its configuration files and could possibly find out whether it is installed or whether it is running uninstalled via PERL5LIB. But as I don't know anything about Prima, that just might not be possible because the C libraries look in fixed special locations. Another thing that might trip you up is that the tests seem to be run without a display and Prima possibly wasn't installed with support to run headless.

I understand that you don't like getting FAIL reports for things that you think are beyond your control and if adapting Prima is out of the question, I would try the following approach to the Image::Match test suite, by having a file 000_prima.t which mainly checks that Prima is available:

#!/usr/bin/perl -w
use strict;
use Test::More;

if (! eval 'use Prima; 1') {
    BAIL_OUT("Prima not installed properly.");
};

ok($Prima::VERSION, "Found Prima version $Prima::VERSION");
[download]

That file will be run as one of the first files and it will stop all testing unless Prima is found to be working well enough for a use Prima;.