Hugo here, I was typing up a response to this last night when my computer died stone dead with a strong whiff of ozone. I'll be offline for a few days until I can get that fixed.

I found that the issue could easily be replicated with something roughly like the below. (This is from memory, I'm in a local hotel to write this.) Please verify and report to the Test::More bug queue.

The problem appears to be occur when a) the '!=' test fails, b) the object(s) being tested have '!=' and '""' overloading but not '0+'. I have not checked if both arguments need to be objects, nor if other comparators are affected.

use Test::More;
package Ov {
  use overload (
    '""' => sub { 'x' },
    '!=' => sub { 0 },
  };
};
cmp_ok(bless({}, 'Ov'), '!=', bless({}, 'Ov'), 'failing test');
done_testing();
[download]