in reply to Guess between UTF8 and Latin1/ISO-8859-1

I tried three options. One using the pack('U0U*",...) solution from perluniintro, second using the regexps suggested by bart and a Encode::decode_utf8() using solution from also from peruniintro. The decode_utf8() is the fastest by far:

my $test = 0; use warnings; use bytes; use Benchmark; use Encode qw(encode_utf8 decode_utf8); my $xml; { my $isUTF = 1; my $sub = sub {$isUTF = 0}; sub byPack { $SIG{__WARN__} = $sub; no warnings 'void'; my @a=unpack( 'U0U*', $xml); delete $SIG{__WARN__}; return $isUTF; } } sub byRegExp { my $bad_utf8 = 0; while($xml =~ /(?=[\x80-\xFF])(?:[\xC0-\xFF][\x80-\xBF]+|(.))/g an +d !$bad_utf8) { $bad_utf8++ if defined $1; } return !$bad_utf8; } sub byDecode { if (decode_utf8($xml)) { return 1 } else { return 0 } } print "OK\n"; open XML, '<test-ok.xml'; $xml = do {local $/; <XML>}; close XML; if ($test) { print "byPack=".byPack()."\n"; print "byRegExp=".byRegExp()."\n"; print "byDecode=".byDecode()."\n"; } else { timethese (10000, { byPack => \&byPack, byRegExp => \&byRegExp, byDecode => \&byDecode, }); } print "BAD\n"; open XML, '<test-bad.xml'; $xml = do {local $/; <XML>}; close XML; if ($test) { print "byPack=".byPack()."\n"; print "byRegExp=".byRegExp()."\n"; print "byDecode=".byDecode()."\n"; } else { timethese (10000, { byPack => \&byPack, byRegExp => \&byRegExp, byDecode => \&byDecode, }); } __END__ OK Benchmark: timing 10000 iterations of byDecode, byPack, byRegExp... byDecode: 0 wallclock secs ( 0.22 usr + 0.00 sys = 0.22 CPU) @ 45 +662.10/s (n=10000) (warning: too few iterations for a reliable count) byPack: 15 wallclock secs (15.17 usr + 0.00 sys = 15.17 CPU) @ 65 +9.11/s (n=10000) byRegExp: 5 wallclock secs ( 4.22 usr + 0.00 sys = 4.22 CPU) @ 23 +70.79/s (n=10000) BAD Benchmark: timing 10000 iterations of byDecode, byPack, byRegExp... byDecode: 0 wallclock secs ( 0.08 usr + 0.00 sys = 0.08 CPU) @ 12 +8205.13/s (n=10000) (warning: too few iterations for a reliable count) byPack: 15 wallclock secs (15.42 usr + 0.00 sys = 15.42 CPU) @ 64 +8.42/s (n=10000) byRegExp: 5 wallclock secs ( 4.25 usr + 0.00 sys = 4.25 CPU) @ 23 +52.94/s (n=10000)

The tests were run with two 4KB XMLs, the bad one had an í character added approximately in the middle.

Jenda
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
   -- Rick Osborne

Edit by castaway: Closed small tag in signature