Welcome to the wonderful world of XML!

I can't figure out exactly what is your original format but I will nevertheless go for the shameless plug:.

<shameless_plug>XML::Twig will happily deal with this problem. get the latest version (3.00) from here and you won't have to bother with entities being dropped.</shameless_plug>

Try playing with this code (with and without the keep_encoding option for example):

#!/bin/perl -w
use strict;
use XML::Twig;

my $t= new XML::Twig( keep_encoding => 1);

{ $/= '';
  while( <DATA>)
    { $t->parse( $_);
      $t->print;
      print "\n";
    }
}

__DATA__
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE doc SYSTEM "dummy"[]>
<doc att="valué ">A document with text in latin1: soupçonné d'être</do
+c>

<?xml version="1.0"?>
<!DOCTYPE doc SYSTEM "dummy"[]>
<doc att="valu&eacute;">A document with text in latin1:soup&ccedil;onn
+&eacute; d'etre</doc>
[download]

Most Perl XML modules are built on top of expat, or XML::Parser which is an interface to expat. Expat is XML parser. It will get your XML (XHTML) document and process its tags and so on. But as XML is fundamentaly based on unicode, expat will convert all your characters to unicode. For this conversion to work properly, you should have valid encoding specified in XML header: <?xml version='1.0' encoding='iso-8859-2'?> This is the primary reason for these odd charaters you encounter. They are utf-8 (8-bit Unicode) representation of non-english characters.

You probably want to avoid this coversion. I have similar problem maybe a year ago, but found no useful solution. XML::Parser has a original_string method which returns character data in original encding, but it wont expand entities. And there is no way to get attributes in original encoding. Best solution around this is to use Unicode::Map8 to map all unicode strings back to their original encodig, but this is terribly slow solution for frequent use.

So I wrote my own poor man's XML parser based on Perl patterns. But it is not a solution, but a hack. If you plan to use XML, use should better move to Unicode completly.

PS: I wonder how XML::Twig implements its keep_encoding option. By forcing expat to behave reasonably or by back conversion to original charset?

XML::Twig uses the original_string method to keep the characters in the original encoding (but then it works only for 1-byte encodings as it uses a regexp to parse the start tag string to extract the tag name and the attributes). In order to track the entities (and not expand them) I use a Default handler that spots them and stores them as a special element.

The latest (still beta) version also comes with a bunch of filters, to convert the UTF-8 back to latin1, html-style text (using HTML::Entities), DOM-style ASCII + character entities or to any other encoding using either the Unicode::Map8 or (even better if the iconv library is installed on your system) Text::Iconv.

Overall using the original_string method, even though it is frowned upon as not being completely kosher is the easiest choice if (IF) you are using a 1-byte encoding. Dealing with the various cases on internal and external entities (depending on whether they are defined at the beginning of the document or in a separate file) is way trickier and entities within attributes are generally a huge pain to deal with using XML::Parser.