Re: TreeBuilder and encoding

..Ive tried explicitly setting STDOUT to utf-8 via binmode..
Actually, your STDOUT should have been set to use ":encoding(UTF-8)" instead of "utf-8". Why?

":utf8" just marks the data as UTF-8 without further checking, while ":encoding(UTF-8)" checks the data for actually being valid UTF-8

this works for me though:

use warnings;
use strict;
use utf8;
use LWP::UserAgent;
use HTML::TreeBuilder;

my $url =
'http://buyingguide.winemag.com/catalog/peju-1998-reserve-cabernet-sau
+vignon-napa-rutherford';

my $browser = LWP::UserAgent->new;

my $re = $browser->get($url);

if ( $re->is_success ) {
    my $tree = HTML::TreeBuilder->new;
    $tree->parse( $re->decoded_content );
    $tree->eof();
    binmode STDOUT, ":encoding(UTF-8)";
    my $review_et = $tree->look_down( 'itemprop', 'reviewBody' );
    my $str = $review_et->as_text;

    #print $str,$/;  # this as works
    print $review_et->as_HTML;
    $tree->delete;
}
else {
    die $re->status_line();
}
[download]

If you tell me, I'll forget.
If you show me, I'll remember.
if you involve me, I'll understand.
--- Author unknown to me

Comment on Re: TreeBuilder and encoding Download Code

Replies are listed 'Best First'.
Re^2: TreeBuilder and encoding by VineMob (Initiate) on Jul 15, 2013 at 13:59 UTC
..Ive tried explicitly setting STDOUT to utf-8 via binmode.. Actually, your STDOUT should have been set to use ":encoding(UTF-8)" instead of "utf-8". Sorry for being imprecise, I had in fact tried both ":encoding(UTF-8)" and "utf-8". this works for me though: hmmm.... it actually doesnt for me. Which suggests it may be a platform issue. Ill try it on a few different boxen and let you know how it works out.	[reply]
Re^3: TreeBuilder and encoding by Khen1950fx (Canon) on Jul 15, 2013 at 23:08 UTC
I think that you are working it a little to hard. There is no `"utf-8"`, but there is `":utf8"`. I always use `":encoding(UTF-8)"`, just to be safe. Here's what I did: If you use the `new_from_url` method, then it will call LWP::UserAgent for you. `#!/usr/bin/perl use strict; use warnings; use HTML::TreeBuilder 5 -weak; my $url = 'http://buyingguide.winemag.com/catalog/peju-1998-reserve-cabern +et-sauvignon-napa-rutherford'; my $tree = HTML::TreeBuilder->new_from_url( $url ); $tree->parse_content( $url ); my $review_et = $tree->look_down('itemprop', 'reviewBody'); binmode STDOUT, ":encoding(UTF-8)"; print $review_et->as_text; $tree->delete;` [download]	[reply] [d/l] [select]