See perldoc Encode under the heading "Handling Malformed Data". The CHECK parameter gives you several ways to handle bad data.
| [reply] |
Thanks for your reply!
So if I do something like this, it should convert any malformed utf-8 to valid utf-8: (?)
#$untrusted-ut8-string = a string that is supposed to contain only utf-8 characters
#i put in the default CHECK value to be explicit
my $octets = encode("utf8", $untrusted-ut8-string, Encode::FB_DEFAULT);
my $perfect-utf-string = decode("utf8", $octets, Encode::FB_DEFAULT);
It seemed to work but I'm always concerned about harmful side effects. Don't want to mess up the actual valid characters
| [reply] |
Yes, that's exactly the sort of thing I had in mind. You may want to experiment with other values of CHECK, to see which fits your repair strategy best.
As for losing data, just be sure not to overwrite the original stuff. Encode should not modify its object string without documenting that.
On a different plane, it would be a good thing to yell at whoever is supplying invalid utf8.
| [reply] |
| [reply] |
This could get dicey, because there are lots of ways for a given byte sequence to violate the various constraints of utf8. When in doubt about how bad the data might be, the best first step would generally be one that just traps and reports the problem -- something like this:
my $untrusted = <>; # (or whatever you use to fetch it)
my $test_utf8;
eval "\$test_utf8 = decode( 'utf8', \$untrusted, Encode::FB_CROAK )";
if ( $@ ) {
warn "Input data is not valid utf8: $@";
# take other appropriate action(s)...
}
It might be worth while to run a script that does just this on the input data, before you pass the data into any other script to actually do something with it, just to see whether the problems are infrequent or rampant. Sometimes they are fairly easy to diagnose, but you may need to be able to track through a hex dump of the relevant bytes and be able to understand both the intention and the nature of how that differs from the reality.
I don't know if it'll help, but a prior discussion about a similar kind of problem came up here: Guess between UTF8 and Latin1/ISO-8859-1 | [reply]
[d/l] |