Re: Encoding horridness

You will also have to make sure that the data you are writing to the XML file has been read properly from your data source and has been properly decoded when reading it.

Ideally you use Encode and decode all data when you read it into your program and encode it when writing it to your output. You have already taken care of encodeing the output, but the input might not be valid UTF-8 or not be recognized by Perl as such.

Assuming that your input data is a file with bytes encoded in Latin-1, you could read/decode the data as

while( <$fh>) {
    my $payload = decode('Latin-1', $_);
};
[download]

For database values, you have the additional fun of finding out as what kind of data/encoding your database actually stores the values.

Comment on Re: Encoding horridness Select or Download Code

Replies are listed 'Best First'.
Re^2: Encoding horridness by Anonymous Monk on Jul 12, 2017 at 14:07 UTC
Good advice to be sure. But since latin-1 is a subset of unicode, isn't `decode('Latin-1', $_)` pretty much a no-op?	[reply]
Re^3: Encoding horridness by Corion (Patriarch) on Jul 12, 2017 at 14:20 UTC
No, because high-bit characters/octets in Latin-1 encode differently as octets in UTF-8, and Perl doesn't know what to do with high-bit characters when writing them.	[reply]
Re^4: Encoding horridness by Anonymous Monk on Jul 12, 2017 at 14:26 UTC
What I'm wondering, though, is if there's ever a situation where `encode('utf8', decode('Latin-1', $_))` [download] produces different output from `encode('utf8', $_)` [download]	[reply] [d/l] [select]
Re^5: Encoding horridness by choroba (Cardinal) on Jul 12, 2017 at 16:51 UTC
Re^6: Encoding horridness by Anonymous Monk on Jul 12, 2017 at 17:55 UTC
Re^3: Encoding horridness by hippo (Archbishop) on Jul 12, 2017 at 14:16 UTC
The OP wants to move from Latin-1 to UTF-8. Latin-1 is not a subset of UTF-8.	[reply]
Re^4: Encoding horridness by Anonymous Monk on Jul 12, 2017 at 14:20 UTC
Yes, and `encode('utf8', decode('Latin-1', $_))` isn't a no-op.	[reply]