I'd give it a little extra tweak:

my %entities = (
    special => 1,
    case    => 1,
);

$xml =~ s/&(?=(\w+);)/exists $entities{$1} ? "&" : die "Unknown en
+tity '$1'"/ge;
[download]

This gives you a fair amount of flexibility in defining your entities, but still leaves you with some way of catching bad ones if they get in. Of course, you could also just make the right side of the conditional simply leave the string untouched, and let XML::Simple catch the error, but this seems unkind. If admittedly kind of cute.