Re^5: text encodings and perl

If the return value is "false", then perl does not know which encoding is really used, so it may use Latin1, or whatever is found working "most of the time"

I'm curious how you came to that conclusion. For any text operation, perl has to assume an encoding. It uses UTF-8 if the utf8 flag is present on the string, and Latin 1 otherwise (assuming you didn't mess with locales). It has no notion of "working" and "most of the time".

I'm well aware of when I need to decode, and when not. And my point was that deciding this question based on the return value of is_utf8 is wrong.

In fact, in your examples perl effectively calls the Encode::decode but with parameter "Latin1" instead of "UTF-8"

It doesn't. Because Latin-1 strings themselves can be perfectly fine text strings.

If you don't believe me, add a warn to the Encode::decode() function of your local perl installation. You'll observe no such call.

Perl 6 - second systems done right

Comment on Re^5: text encodings and perl

Replies are listed 'Best First'.
Re^6: text encodings and perl by andal (Hermit) on Nov 15, 2010 at 12:21 UTC
I'm curious how you came to that conclusion. For any text operation, perl has to assume an encoding. It uses UTF-8 if the utf8 flag is present on the string, and Latin 1 otherwise (assuming you didn't mess with locales). It has no notion of "working" and "most of the time". Looks like we are talking about completely different things. I can not find anywhere where my words contradict to yours. Yes, perl has to assume an encoding. I'm talking only about the fact that the developer shall make sure that the assumed encoding is correct. For this he/she shall use the Encode::decode methods, or the conveniences of perl like binmode or -CS switch or whatever. It doesn't. Because Latin-1 strings themselves can be perfectly fine text strings. Sorry, but do you understand the difference between words "effectively calls" and "really calls"? Of course perl does not call this function. It just assumes that the sequence of octets it has is encoded as Latin1. If it really is then everything works. But if it is not, then nothing works. I really don't understand where do you find in my words, that strings without utf8 flag are not text strings. They are. It's just that perl works with them as Latin1 strings, which is incorrect if your input is not Latin1. That's all. Nothing else.	[reply]

Replies are listed 'Best First'.

Re^6: text encodings and perl
by andal (Hermit) on Nov 15, 2010 at 12:21 UTC

I'm curious how you came to that conclusion. For any text operation, perl has to assume an encoding. It uses UTF-8 if the utf8 flag is present on the string, and Latin 1 otherwise (assuming you didn't mess with locales). It has no notion of "working" and "most of the time".

Looks like we are talking about completely different things. I can not find anywhere where my words contradict to yours. Yes, perl has to assume an encoding. I'm talking only about the fact that the developer shall make sure that the assumed encoding is correct. For this he/she shall use the Encode::decode methods, or the conveniences of perl like binmode or -CS switch or whatever.

It doesn't. Because Latin-1 strings themselves can be perfectly fine text strings.

Sorry, but do you understand the difference between words "effectively calls" and "really calls"? Of course perl does not call this function. It just assumes that the sequence of octets it has is encoded as Latin1. If it really is then everything works. But if it is not, then nothing works.

I really don't understand where do you find in my words, that strings without utf8 flag are not text strings. They are. It's just that perl works with them as Latin1 strings, which is incorrect if your input is not Latin1. That's all. Nothing else.

[reply]