Entities confuse encoding in XML::Simple

gizzlon has asked for the wisdom of the Perl Monks concerning the following question:

Hello I'm using XML::Simple to read a latin1 (iso-8859-1) xml file and everything is great until it encounters some &#xxx style entities. Strings containing those entities seem to be double encoded or some other weirdness:

 <?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<foo>
<p>foo</p>
<p>foo with an &#8211;</p>
<p>some latin1 encoded chars: жше ЖШЕ</p>
<p>same, but this time whith an &#8211; .. жше ЖШЕ</p> 
<p>same, but this thime with an &#8221; instead .. : жше ЖШЕ</p>
</foo>
[download]

Read by this script:

my $xmldata = XMLin( $ARGV[0], ForceArray=>1, KeyAttr=>{meta=>"name"} 
+, SuppressEmpty=>"") or die "Could not parse xml data: $!";

foreach my $f ( @{$xmldata->{'p'} } ) {
  print $f;
  print "\n";
}

print Dumper($xmldata);
[download]

Produces:

 ./test2.pl foo.xml 
foo
Wide character in print at ./test2.pl line 12.
foo with an –
some latin1 encoded chars: жше ЖШЕ
Wide character in print at ./test2.pl line 12.
same, but this time whith an – .. Г¦ГёГҐ Г&#134;Г&#152;Г&#133;
Wide character in print at ./test2.pl line 12.
same, but this thime with an ” instead .. : Г¦ГёГҐ Г&#134;Г&#152;Г&#13
+3;
$VAR1 = {
          'p' => [
                   'foo',
                   "foo with an \x{2013}",
                   'some latin1 encoded chars: жше ЖШЕ',
                   "same, but this time whith an \x{2013} .. \x{c3}\x{
+a6}\x{c3}\x{b8}\x{c3}\x{a5} \x{c3}\x{86}\x{c3}\x{98}\x{c3}\x{85}",
                   "same, but this thime with an \x{201d} instead .. :
+ \x{c3}\x{a6}\x{c3}\x{b8}\x{c3}\x{a5} \x{c3}\x{86}\x{c3}\x{98}\x{c3}\
+x{85}"
                 ]
        };
[download]

Looks like its double encoded?
Any ideas?

Thanx

Comment on Entities confuse encoding in XML::Simple Select or Download Code

Replies are listed 'Best First'.
Re: Entities confuse encoding in XML::Simple by moritz (Cardinal) on Jan 03, 2008 at 11:30 UTC
Maybe you have to to set up your environment a bit better: `binmode STDOUT, ':encdoding(UTF-8)'; # or whatever your terminal uses` [download]	[reply] [d/l]
Re^2: Entities confuse encoding in XML::Simple by Anonymous Monk on Jan 03, 2008 at 13:17 UTC
Even if it somehow had the wrong encoding, isn't it strange that some of the output is correct and some is not? Anyway, the terminal is utf8 and I was surprised to see that `binmode STDOUT, ':encoding(UTF-8)'` actually made it worse: foo foo with an – some latin1 encoded chars: Г¦ГёГҐ ГГГ same, but this time whith an – .. Г¦ГёГҐ ГГГ same, but this thime with an ” instead .. : Г¦ГёГҐ ГГГ&#13 +3; $VAR1 = { 'p' => [ 'foo', "foo with an \x{2013}", 'some latin1 encoded chars: Г¦ГёГҐ ГГГ& +#133;', "same, but this time whith an \x{2013} .. \x{c3}\x{ +a6}\x{c3}\x{b8}\x{c3}\x{a5} \x{c3}\x{86}\x{c3}\x{98}\x{c3}\x{85}", "same, but this thime with an \x{201d} instead .. : + \x{c3}\x{a6}\x{c3}\x{b8}\x{c3}\x{a5} \x{c3}\x{86}\x{c3}\x{98}\x{c3}\ +x{85}" ] }; [download]	[reply] [d/l] [select]
Re^3: Entities confuse encoding in XML::Simple by Anonymous Monk on Jan 09, 2008 at 14:29 UTC
Did you find a solution? I'm having the same problem as you.	[reply]
Re^4: Entities confuse encoding in XML::Simple by gizzlon (Initiate) on Jan 09, 2008 at 18:02 UTC