in reply to UTF-8 Validation
It might be worth while to run a script that does just this on the input data, before you pass the data into any other script to actually do something with it, just to see whether the problems are infrequent or rampant. Sometimes they are fairly easy to diagnose, but you may need to be able to track through a hex dump of the relevant bytes and be able to understand both the intention and the nature of how that differs from the reality.my $untrusted = <>; # (or whatever you use to fetch it) my $test_utf8; eval "\$test_utf8 = decode( 'utf8', \$untrusted, Encode::FB_CROAK )"; if ( $@ ) { warn "Input data is not valid utf8: $@"; # take other appropriate action(s)... }
I don't know if it'll help, but a prior discussion about a similar kind of problem came up here: Guess between UTF8 and Latin1/ISO-8859-1
|
|---|