jalewis2:

XML::Simple will also take a string instead of a filehandle, so you could slurp in the file, do a quick patch of the XML header to correct the encoding specification and then pass the text to XML::Simple.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

The issue is that the xml claims to be UTF8, but it has ISO-8859-1 characters. I've successfully converted it from the command line, but when I try to put the gzip, conversion and processing together, it breaks.

It didn't dawn on me that changing the xml header might tell XML::Simple to read the file differently. Is that what you're suggesting?

jalewis2:

Exactly. I'm thinking that you may be able to update the xml header to add the proper encoding attribute, something like <?xml version='1.0' encoding='ISO-8859-1' ?> as described in the Section 4.3 "Parsed Entities" in the XML specification. I don't know if this trick will work, as I've never used it. But hopefully, it'll get you past your hurdle. I'm not very knowledgeable about XML, so

Another link I use when trying to debug XML problems is: The W3C XML Markup Validation Service. That's where I point people to when they tell me I'm wrong about their file being incorrect.

...roboticus

When your only tool is a hammerJava, all problems look like your thumbXML.

You could open up each file, detect a sampling of the octet stream, then properly encode the non-utf8 ones.

#!/usr/bin/perl
use warnings;
use strict;
use Encode::Detect::Detector;


my $octets = "\x{4f60}\x{597d}\x{4e16}\x{754c}"; 
my $charset = Encode::Detect::Detector::detect($octets);
print "$charset\n";

$octets = "\x82\xb7\x82\xb2\x82\xa2\x82\xcc\x82\xdd\x82\xc2"; 
$charset = Encode::Detect::Detector::detect($octets);
print "$charset\n";


$octets = "\x{805a}\x{5408}\x{6216}\x{8be6}\x{7ec6}"; 
$charset = Encode::Detect::Detector::detect($octets);
print "$charset\n";
[download]