It ignores those files when creating the distribution, not when creating the MANIFEST. I create my MANIFEST by hand, so that I know exactly what should be distributed. I'm also, from what I understand, in the minority of CPAN maintainers who do this when using MB.
Test using "./Build dist" and see what gets put in after you hand-build your MANIFEST.
My criteria for good software:
Does it work?
Can someone else come in, make a change, and be reasonably certain no bugs were introduced?