Thanks for your reply!
So if I do something like this, it should convert any malformed utf-8 to valid utf-8: (?)
#$untrusted-ut8-string = a string that is supposed to contain only utf-8 characters
#i put in the default CHECK value to be explicit
my $octets = encode("utf8", $untrusted-ut8-string, Encode::FB_DEFAULT);
my $perfect-utf-string = decode("utf8", $octets, Encode::FB_DEFAULT);
It seemed to work but I'm always concerned about harmful side effects. Don't want to mess up the actual valid characters
| [reply] |
Yes, that's exactly the sort of thing I had in mind. You may want to experiment with other values of CHECK, to see which fits your repair strategy best.
As for losing data, just be sure not to overwrite the original stuff. Encode should not modify its object string without documenting that.
On a different plane, it would be a good thing to yell at whoever is supplying invalid utf8.
| [reply] |