Yes it worries me too. sv_unmagic() on '~' removes all '~' magic entries currently (Which is why the current implemenation does not call it). I wonder if there would be any way or (where) to determine what modules exist that use internal magic. I would think there are not too many. The only one I know of is Devel::WeakRef and Pixie.
I don't mind taking my own magic entry but worry about clashes with future versions of Perl 5.* and of course other modules developers.