Re^3: Difficulty with UTF-8 and file contents

Thanks, everybody.

It came down to some gradual, one-step-at-a-time debugging combined with your advice above.

The wrong code which caused the problem:

    my $xhtml = HTML::TreeBuilder::XPath->new;
    $xhtml->implicit_tags(1);
    $xhtml->parse_file($file)
        or die("Could not parse '$file' : $!\n");
[download]

The code which prevented the mutilation of the data:

    . . .
    use open qw/:std :utf8/;
    . . .

    my $xhtml = HTML::TreeBuilder::XPath->new;
    $xhtml->implicit_tags(1);
    my $filehandle;
    open ($filehandle, "<", $file)
        or die("Could not open file '$file' : error: $!\n");
    $xhtml->parse_file($filehandle)
        or die("Could not parse file handle for '$file' : $!\n");
[download]

So if I guess right, the use of a file handle which I have opened myself under the influence of the use open qw/:std :utf8/; pragma forced the data going into HTML::TreeBuilder::XPath to be read as UTF-8?

Comment on Re^3: Difficulty with UTF-8 and file contents Select or Download Code

Replies are listed 'Best First'.
Re^4: Difficulty with UTF-8 and file contents by haukex (Archbishop) on Apr 13, 2020 at 17:26 UTC
So if I guess right, the use of a file handle which I have opened myself under the influence of the `use open qw/:std :utf8/;` pragma forced the data going into HTML::TreeBuilder::XPath to be read as UTF-8? Yes, that's correct. Note the documentation of `parse_file` in HTML::TreeBuilder: ... When you pass a filename to `parse_file`, HTML::Parser opens it in binary mode, which means it's interpreted as Latin-1 (ISO-8859-1). If the file is in another encoding, like UTF-8 or UTF-16, this will not do the right thing. One solution is to open the file yourself using the proper `:encoding` layer, and pass the filehandle to `parse_file`. ...	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^4: Difficulty with UTF-8 and file contents
by haukex (Archbishop) on Apr 13, 2020 at 17:26 UTC

So if I guess right, the use of a file handle which I have opened myself under the influence of the use open qw/:std :utf8/; pragma forced the data going into HTML::TreeBuilder::XPath to be read as UTF-8?

Yes, that's correct. Note the documentation of parse_file in HTML::TreeBuilder:

... When you pass a filename to parse_file, HTML::Parser opens it in binary mode, which means it's interpreted as Latin-1 (ISO-8859-1). If the file is in another encoding, like UTF-8 or UTF-16, this will not do the right thing. One solution is to open the file yourself using the proper :encoding layer, and pass the filehandle to parse_file. ...

[reply]
[d/l]
[select]