certain functions (e.g., lc) change their behavior depending on how the flag is set.

Yes, this is indeed the fly in the ointment. As far as I know such cases are documented - and in this case at least the documentation describes mechanisms that force it to behave one way or another independent of the UTF8 flag (eg use bytes versus use feature 'unicode_strings').

Those aspects of Perl that requires you to know the state of the UTF8 flag are collectively known as "the Unicode bug", and there is more detail in a section devoted to this in perlunicode.

Those aspects of Perl that requires you to know the state of the UTF8 flag

But I'm not asking "what is the state of the UTF8 flag." I'm asking, "Does a given operation preserve the state?"

  $str2 = $str;
  $str3 = sprintf ("str is %s", $str);
  @words = split(/ /, $str);
[download]

Do $str2, $str3, and $words[2] have the same flag value as $str? Does it depend on other factors? Is it undefined? (I suppose since it's intentionally undocumented, it's at least theoretically undefined.)

I'd need to (laboriously) check the source for chapter and verse, but as far as I remember in all the obvious cases when any of the inputs have UTF8 on, the output will too.

Here's an example commonly used in perl's tests to create a UTF8-flagged string by appending a flagged zero-length string:

% perl -MDevel::Peek -wle '
  $x="\x{100}"; Dump($x);
  chop $x;      Dump($x);
  $y = "foo";   Dump($y);
  $y .= $x;     Dump($y)
' 2>&1 | grep FLAGS
  FLAGS = (POK,IsCOW,pPOK,UTF8)
  FLAGS = (POK,pPOK,UTF8)
  FLAGS = (POK,IsCOW,pPOK)
  FLAGS = (POK,pPOK,UTF8)
%
[download]

Your examples certainly all appear to propagate the flag. However substr() appears to propagate it only if the resulting substring source string has characters above 0x7f: I have no idea why that appears to be an exception. And I also do not know of any guarantee that any of these behaviours will be retained in future perl versions (though I think it is hugely unlikely that the steps involved in the code above will change, due to its widespread use in core).

Update: from perl source, it appears substr returns a UTF8_off result if the source string has byte length and character length the same.

If that's the intent

It is. Code that behaves differently based on the internal storage format is said to suffer from The Unicode Bug.

it doesn't always work in practice

True. Notably, the operators that accept file names. And of course, some XS modules.

utf8::upgrade and utf8::downgrade can be used to work around these bugs.

certain functions (e.g., lc) change their behavior depending on how the flag is set.

lc, uc and the regex engine were fixed in 5.14, released in 2011 (12 years ago).

To get the fix, you need to use use v5.14;, or use feature qw( unicode_strings ); more specifically.

(The feature actually appeared in 5.12, but it didn't fix as many things in 5.12 as in 5.14, so I pretend it was added in 5.14.)

Could you please provide an example where lc behaves different, depending on the flag?

As far as I know lc will simply preserve the flag of the input (I am not sure whether this holds on EBCDIC platforms).

The opposite function, uc, is known to set the flag for a (non-flagged) input of chr 0xFF or 'ÿ': Its uppercase equivalent 'Ÿ' is not present in ISO-8859-1, but taken from the Unicode block Latin Extended-A.

Not sure about lc(), but here's another case where the closely-related uc() behaves differently:

$ascii = "\x{df}";
chop($utfer = "\x{100}");
$utf = $ascii . $utfer;
print uc($_) for ($ascii, $utf);
[download]

As a Unicode codepoint, "\x{df}" is interpreted as the lowercase German "es-zed" character (ß), which uppercases to "SS". As an ASCII codepoint it is seen as a non-word character, and does not change.

This is a rare case where changing the case of a string also changes its length.

As an ASCII codepoint

Nitpick: it isn't ASCII. I suspect you meant either ISO-8859-1 or Latin-1 or non-Unicode instead of ASCII which has a highest codepoint of \x{7f}.

---

🦛

Ah, interesting. I missed that because it behaves differently depending on the use of features:

haj@vdesktop:~$ perl -M5.010 -C -e 'print uc chr 0xdf, "\n"'
ß
haj@vdesktop:~$ perl -M5.012 -C -e 'print uc chr 0xdf, "\n"'
SS
[download]

That's the point you seem to be missing.

The function length must report different numbers of characters, if 2-4 bytes are supposed to represent a unicode entity because of the utf8-flag. Same for other functions.

Otherwise please be more specific about what lc does wrongly...

The function length() always reports the number of characters in the string: you do not need to know whether the UTF8 flag is set on your string to understand what it will do.

The function lc() on the other hand will give different results for the same string (ie a string consisting of the same characters) depending on whether the UTF8 flag is set or not. As such it is an example of the Unicode bug in action.

This is not necessarily wrong - it is after all documented behaviour. But it does mean that, despite the intent, the programmer needs to know how the UTF8 flag will have been set to correctly predict the behaviour of lc() on strings containing certain characters.

I'm puzzled.

Of course length can also give you different results if you change the UTF8 flag.

Cheers Rolf
_{(addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
Wikisyntax for the Monastery}

Otherwise please be more specific about what lc does wrongly...

As far as I know it does nothing wrongly. It just does things differently depending on previous operations whose effects are not fully documented.

To answer your main question, I would be very surprised if there where normal cases where the UTF8-Flag isn't preserved when passing around. So no need to document the obvious.

Without UTF8-Flag it's a octet-stream and all string commands will treat every single byte (for backward compatibility) as (some) ASCII character.

Otherwise it's a "character-string" and Perl will use the internal UTF8 format to map one or more bytes to characters

Easy.

Cheers Rolf
_{(addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
Wikisyntax for the Monastery}