I've got a fairly old script that uses XML::XPath to parse an XML file that includes personal names with accented letters, e.g.,

<?xml version="1.0" encoding="UTF-8"?>
<contrib contrib-type="author">
    <name>
        <surname>San José Estépar</surname>
        <given-names>Raul</given-names>
    </name>
</contrib>

[download]

The names are assigned variables and then inserted into a MySQL database. Later, a different script collects the names from the database and writes them to a new XML file.

The problem (which may have been going on for a while and has only just been reported) is that the accented letters aren't surviving the journey. When I view the names in the database (using Sequel Pro 1.1 for OS X), I see the correct characters. But when the names are written to a new XML file, the characters appear as question marks (viewing the XML file in Oxygen XML Editor 17.0 for OS X or just via the Unix 'more' command from the OS X Terminal app).

I'm pretty confident that the source XML file is OK. It has the explicit XML declaration (as in the sample above) including 'encoding="UTF-8"', and it displays correctly any way I view it (from the Unix command-line, from Oxygen, etc.).

I have also verified that reading the XML file and writing it out again like so:

my $xml;
open (SOURCE, $source_file)
    or die "ERROR: Cannot open filehandle to read $source_file: $!\n";
{
    local $/;
    $xml = <SOURCE>;
}
close(SOURCE)
    or die "ERROR: Cannot close filehandle on $source_file: $!\n";
print $xml;

[download]

preserves the characters. The problem occurs when I parse the file content like so:

my $xp = XML::XPath->new(xml => $xml)
    or die "ERROR: XML::XPath cannot parse target file: $!\n";

my $contrib_nodeset =
    $xp->find('/contrib[@contrib-type="author"]');
foreach my $contrib_node ($contrib_nodeset->get_nodelist) {
    my $given_names = $xp->find('./name/given-names', $contrib_node);
    my $surname     = $xp->find('./name/surname',     $contrib_node);
    print "CONTRIB: $given_names $surname\n";
}

[download]

I confess I haven't had to deal much with character encoding before, so I'd be grateful for any advice on how to troubleshoot this problem.

In reply to XML::XPath and character encoding by mboudreau

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.