To test something, you have to test it against something. Like, a specification.

Currently there is none, so I see no point in testing. But IMHO that should change.

I find this lack of computer readable meta information on CPAN quite sad, because it prevents many useful applications (for example I'd like to write a CPAN mirror that automatically builds debian packages from CPAN packages, but currently there's not even an consensus how dependencies on non-perl libraries should be declared).

So maybe the right thing to do is to specify an option in META.yml which specifies the Changelog format, so if a package includes that option the test is run, if not you should only test that the file exists and is not empty.