XML::Twig, HTML::Table, and wide characters

eff_i_g has asked for the wisdom of the Perl Monks concerning the following question:

Monks:

I'm using HTML::Table to build—you guessed it—an HTML table, and then incorporate that into XML::Twig.

As soon as I put these two together I see "Wide character in print..." and XML::Twig (apparently) no longer honors output_text_filter.

The code follows, as well as the results explained in the code:

use warnings;
use strict;
use Encode;
use XML::Twig;
use HTML::Table;

undef $/;
my $html = <DATA>;

my $twig = XML::Twig->new(
    input_filter => sub {
        my $txt = shift;
        return decode('Windows-1252', $txt);
    },
    output_text_filter => 'safe_hex',
    pretty_print => 'indented',
    twig_print_outside_roots => 1,
    twig_roots => {
        'h1' => sub {
            ### If 1:
            ###  - "Wide character in print..."
            ###  - TM is *not* converted to &#x2122; by output_text_fi
+lter
            ### If 0:
            ###  - No character complaints
            ###  - TM *is* converted
            if (1) {
                my $new_elt = XML::Twig::Elt->new('div');
                my $table = HTML::Table->new([[1,2]]);
                $new_elt->set_inner_html($table->getTable());
                $new_elt->paste(before => $_);
            }
            $_->flush();
        }
    },

);
$twig->parse_html($html);

__DATA__
<html>
<body>
    <h1>Such and Such™</h1>
</body>
</html>
[download]

The result with 1:

<?xml version="1.0" encoding="iso-8859-1"?><html><head></head><body>
    
  <div>
    <table>
      <tbody>
        <tr>
          <td>1</td>
          <td>2</td>
        </tr>
      </tbody>
    </table>
  </div>
Wide character in print at /usr/local2/lib/perl_aps/XML/Twig.pm line 7
+662, <DATA> chunk 1.
  <h1>Such and Suchâ¢</h1>
</body>


</html>
[download]

The result with 0:

<?xml version="1.0" encoding="iso-8859-1"?><html><head></head><body>
    
  <h1>Such and Such&#x2122;</h1>
</body>


</html>
[download]

Q: Why don't I/How can I get:

<?xml version="1.0" encoding="iso-8859-1"?><html><head></head><body>
    
  <div>
    <table>
      <tbody>
        <tr>
          <td>1</td>
          <td>2</td>
        </tr>
      </tbody>
    </table>
  </div>
  <h1>Such and Such&#x2122;</h1>
</body>


</html>
[download]

Thanks, this had been puzzling me all afternoon!

Comment on XML::Twig, HTML::Table, and wide characters Select or Download Code

Replies are listed 'Best First'.
Re: XML::Twig, HTML::Table, and wide characters by ikegami (Patriarch) on Jan 15, 2009 at 22:37 UTC
An extra tidbit: HTML::Table is a red herring. `my $new_elt = XML::Twig::Elt->new('div'); $new_elt->set_inner_html(''); $new_elt->paste(before => $_);` [download] is enough to trigger the problem, but the following isn't `my $new_elt = XML::Twig::Elt->new('div'); #$new_elt->set_inner_html(''); $new_elt->paste(before => $_);` [download]	[reply] [d/l] [select]
Re: XML::Twig, HTML::Table, and wide characters by mirod (Canon) on Jan 16, 2009 at 14:32 UTC
Humm.. I am a bit busy at the moment, so I haven't had much time to look deep into this. Encoding problems are often a pain. My guess is that the problem is with the `parse_html`. It uses HTML::TreeBuilder to turn the HTML into XML, as does `set_inner_html` BTW. At this point I am not sure of the encodings of the various strings in the twig. The only thing I can think of: could you try doing the `decode` before parsing the html? This way you would be working in utf-8?	[reply] [d/l] [select]
Re^2: XML::Twig, HTML::Table, and wide characters by eff_i_g (Curate) on Jan 16, 2009 at 15:58 UTC
Like this? I get the same result: `my $new_elt = XML::Twig::Elt->new('div'); my $table = HTML::Table->new([[1,2]]); my $content = decode('iso-8859-1', $table->getTable()); $new_elt->set_inner_html($content); $new_elt->paste(before => $_);` [download] Something else to note: If I remove the "TM" from the original XML, everything works; however, as long as it is present, it upsets `set_inner_(?:ht\|x)ml`.	[reply] [d/l] [select]
Re: XML::Twig, HTML::Table, and wide characters by eff_i_g (Curate) on Jan 16, 2009 at 18:27 UTC
Until more is understood, I've come up with the following work-around: `my $new_elt = XML::Twig::Elt->new('div'); my $table = HTML::Table->new([[1,2]]); $new_elt->set_inner_html($table->getTable()); $new_elt->paste(before => $_); ### Required: reinforce the output filter. $_[0]->set_output_filter('safe_hex');` [download]	[reply] [d/l]