Sorry to reply to myself, but I've noticed that the HARNESS_PERL_SWITCHES variable may be useful for this after all, because the -X command line option overrides the -w option and disables all warnings. Just do the following:

HARNESS_PERL_SWITCHES=-X make test
[download]

But if you still wanted to keep the lexical warnings this won't do what you want, because it disables them as well.