Could it be because I am generating the warnings by printing directly to STDERR

Yes:

use strict;
use warnings;

use Test::More tests => 2;
use Test::Warn;

# Your code which gives an expected warning
sub foo {
    warn "Stripe Webhook Error: Invalid Stripe Signature\n";
}

sub bar {
    STDERR->print ("Stripe Webhook Error: Invalid Stripe Signature\n")
+;
}

# Test your code
warning_is { foo (); } "Stripe Webhook Error: Invalid Stripe Signature
+\n",
    'Invalid sig warning issued';
warning_is { bar (); } "Stripe Webhook Error: Invalid Stripe Signature
+\n",
    'Invalid sig STDERR print issued';
[download]

So don't do that. :-)

I came across this very interesting thread which reminded me why I didn't use warn - I don't want line numbers in the warnings from the module!

using warn instead or print STDERR?

The module does have the option to suppress or override warnings. I could use that but wanted to include the error situations in the tests.

I don't want line numbers in the warnings from the module

All you need to do is append a newline:

#!/usr/bin/env perl
use strict;
use warnings;

warn "foo";
warn "bar\n";
[download]

$ perl warns.pl
foo at warns.pl line 5.
bar
$
[download]

---

🦛

So don't do that. :-)

Too late...

I've just sent the latest version off to CPAN to try to deal with the perplexing issue of Test failing during CPAN Testing, which is more urgent than suppressing output to STDERR. That will nave to wait for the next release!