Assuming the input file is really latin-1, you can do a couple of things:

• Add an XML declaration with an encoding attribute to the XML file - <?xml encoding="iso-8859-1"?> - that should make sure the XML parser will do the right thing.
• Convert the XML to utf8, which you've tried but you got it backwards. The correct statement is $xml = decode('iso-8859-1', $xml);
• Figure out some other way to signal the real encoding to XML::Simple. Not sure if you can.