It's failing for three reasons.

• Your first test throws an exception.
• You OS test fails.
• BAIL_OUT causes the process to exit with a non-successful return.

If you had had more tests in the file, those would fail too by virtue of your early bail out.

To skip all tests, you want the skip_all functionality.

use Test::More;

BEGIN {
   if ($^O =~ /^(?:MSWin|VMS|dos|MacOS|os2|epoc|cygwin)/i) {
      plan skip_all => "OS unsupported. It may or may not work, but yo
+u won't know since the tests are being skipped. Use at your own risk"
+;
   } else {
      plan tests => 1;
   }

   use_ok('Term::Menu::Hierarchical');
}
[download]

Or maybe you're really trying to do something other than confuse would-be users.

• Are you trying to prevent the installation of the module on those platforms? You'll need to muck around with Makefile.PL.

  if ($^O =~ /^(?:MSWin|VMS|dos|MacOS|os2|epoc|cygwin)) {
      die "OS unsupported\n";
  }
  [download]

• Are you trying to suppress the emails? Setup a mail filter.

* Your first test throws an exception. * You OS test fails. * BAIL_OUT causes the process to exit with a non-successful return.

As I understood it, that's what I'm supposed to do. According to the "Notes For CPAN Authors" page, in the ""How can I indicate that my distribution only works on a particular operating system?" section, it says

While it isn't a very elegant solution, the recommend approach is to e
+ither die in the Makefile.PL or Build.PL (or BAIL_OUT in a test file)
+ with one of the following messages:

  • No support for OS
  • OS unsupported

CPAN Testers tools will look for one of those phrases and will send an
+ NA (Not Available) report for that platform.
[download]

That's what I'm doing. 'make test' works fine on my system as well as a NetBSD system that I can access; if I change the above OS test to include Linux, it bails properly at that time.

Or maybe you're really trying to do something other than confuse would-be users.

That's unnecessarily and pointlessly unkind as well as somewhat nonsensical. What does a test, which is supposed to prevent installation on non-compliant OSes, have to do with "confusing users"? It explicitly bails on systems that are not supported, which would prevent confusion.

My point is that is should fail on a Windows system, but it should not result in a 'FAIL'ed test from CPANtesters (which, in theory at least, should bail on testing any of the unsupported OSes and test the ones that don't abort.) I am NOT asking "how do I make this not fail on Windows". I am asking "how do I tell CPANtesters that it should not test on Windows, etc. platforms?"

-- 
Education is not the filling of a pail, but the lighting of a fire.
 -- W. B. Yeats

As I understood it, that's what I'm supposed to do.

I didn't know that. That means you can ignore the third bullet, and maybe the second one too. The first one still applies, though. If you don't reach BAIL_OUT, how can it do what it should do?

What does a test, which is supposed to prevent installation on non-compliant OSes, have to do with "confusing users"?

That's not what the code in question does at all. It silences the test failures.

You weren't very clear about what you wanted to do in your original post. It looked to me like you were trying to skip all tests, but I knew I could have guessed wrong. That's why I provided alternative answers. Based on what you say now, the relevant answer is

You'll need to muck around with Makefile.PL. if ($^O =~ /^(?:MSWin|VMS|dos|MacOS|os2|epoc|cygwin)) { die "OS unsupported\n"; } [download]