Greetings honorable monks!

I have a problem where XML parsing returns an apparently malformed utf8 string. I am attaching an example script, complete with relevant data, and a reference to the URL where this data is taken from (the XML is generated by Yahoo). My questions are:

• How can I detect that there is something wrong with the string? Encode::is_utf8 ($str, 1) is supposed to be false (it is supposed to fail the check).
• If I detect a malformed string how can I sanitize it? For instance how can I replace the offending byte sequence with a single '?' or something?

Thank you for your help!

use warnings;
use strict;

use Encode;
use HTML::Entities;
use XML::Twig;

my $objinfo_parser = XML::Twig->new (
    twig_handlers => { 'Product[@Id = "most3usb20on"]' => \&_test_hand
+ler }
);

$objinfo_parser->parse (decode_entities (join '', <DATA>) );
#$objinfo_parser->parseurl ('http://www.3btech.net/objinfo.xml');

sub _test_handler {
    my ($twig, $elt) = @_;
    
    my $str = substr ($elt->field ('Description'), -12);

    print "The string below passes a utf8 well-formedness test, why?\n
+" 
        if Encode::is_utf8 ($str, 1);
    
    print "$str\n\n";
    for (unpack ('U*', $str)) {
        printf "0x%X\n", $_;
    }
    
    print "\n";
    exit;
}

# The xml snippet below is taken directly from the above url
# (the url will take about 15s to download and parse)
# HTML encoded to preserve offending characters

__DATA__
<?xml version="1.0" encoding="UTF-8"?>
    <Product Id="most3usb20on">
      <Description>Moving Star 3.5" USB 2.0 One Button Backup Aluminum
+ Hard Drive Enclosure &#xC2;&#x96; Black</Description>
    </Product>
[download]

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.