Omit your typemap from the MANIFEST, and generate it directly from crap.pod in your Makefile.PL or Build.PL script.

open my $in,   '<', 'crap.pod' or die "moan crap $!\n";
open my $tmap, '>', 'typemap'  or die "typemap $!\n";
while (<$in>) {
    print $tmap $_ if /^=head1/ .. /=^cut/ and /^[^=]/;
}
close $in;
close $tmap;
[download]

Remember that xsubpp will be run much later, and won't care that typemap was created only a few seconds ago, and did not, in fact, come with the original tarball. Just make sure it renders as you want in POD, and remove any POD hackery as you write it out to the typemap file.

Or go the other way: have a POD stub file, and merge in the contents of typemap, appropriately munged.