Converting to latin-1 only works if the characters used are actually in the latin-1 character set.

LEFT DOUBLE QUOTATION MARK and EN DASH only look sort of like " and - but they're not the same characters. And you should get a warning trying to convert them

Using numeric entities should work, but I wonder why you're not just encoding the XML file as UTF-8.