I'm trying to properly handle UTF-8 characters scraped from a web page using the Web::Scraper module. Here is some sample code that simulates a snippet from a utf-8 encoded web page with an ellipsis character: 

use warnings;
use strict;
use utf8;
use Data::Dumper qw(Dumper);
use Web::Scraper::LibXML;

my $scraper = scraper {
        process '//p[contains(@class, "myClass")]',
            'paragraph' => 'TEXT';
    };

print Dumper $scraper->scrape(\<<'END_HTML');
<p class="myClass">…</p>
END_HTML

__END__

[download]

This script outputs: 

$VAR1 = {                                                         
          'paragraph' => "\x{2026}"                                
        };

[download]

I've tried various methods of encoding the input and/or decoding the output of the print statement using different methods and modules but I'm having no luck figuring out how to properly handle these characters so the utf8 characters are generated as they appear in the original source code. As usual, character encoding is confusing the hell out of me. The Web::Scraper manual says "If you pass URI or HTTP::Response object, Web::Scraper will automatically guesses the encoding of the content by looking at Content-Type headers and META tags. Otherwise you need to decode the HTML to Unicode before passing it to scrape method." Can someone please point me in the right direction?

$PM = "Perl Monk's";
$MCF = "Most Clueless Friar Abbot Bishop Pontiff Deacon Curate Priest Vicar";
$nysus = $PM . ' ' . $MCF;
Click here if you love Perl Monks

In reply to Handling utf-8 characters when scraping by nysus

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.