To answer your main question, I would be very surprised if there where normal cases where the UTF8-Flag isn't preserved when passing around. So no need to document the obvious.
Without UTF8-Flag it's a octet-stream and all string commands will treat every single byte (for backward compatibility) as (some) ASCII character.
Otherwise it's a "character-string" and Perl will use the internal UTF8 format to map one or more bytes to characters
Easy.
| [reply] |
Without UTF8-Flag it's a octet-stream and all string commands will treat every single byte (for backward compatibility) as (some) ASCII character
Nitpick: Single bytes also work in the 128-255 range, so it is rather ISO-8859-1 than ASCII. For example, an (ä) encoded as chr 0xE4 matches qr/\w/, according to its unicode property.
| [reply] |
> > Without UTF8-Flag it's a octet-stream and all string commands will treat every single byte (for backward compatibility) as (some) ASCII character
> Nitpick: Single bytes also work in the 128-255 range, so it is rather ISO-8859-1 than ASCII. For example, an (ä) encoded as chr 0xE4 matches qr/\w/, according to its unicode property.
I can't reproduce this, from what I see is \w defaulting to pure ASCII
C:\Users\rolflangsdorf>perl
$a=chr(0xE4); print "$a matched \\w" if $a =~/^\w/;
__END__
C:\Users\rolflangsdorf>perl -v
This is perl 5, version 32, subversion 1 (v5.32.1) built for MSWin32-x
+64-multi-thread
| [reply]
[d/l]
[select] |
| [reply] |
I would be very surprised if there where normal cases where the UTF8-Flag isn't preserved when passing around. So no need to document the obvious.
It sounds like you've confused "expected" with "obvious."
"Widget can contain any ASCII character. This includes the semicolon." That second sentence is obvious—it's an easily deducible consequence of the first—so it need not (and should not) be stated.
Conversely, you can expect things to happen a certain way, but software can sometimes defy your expectation—as hv's remark below about substr demonstrates. That doesn't mean the software is misbehaving; it just means your expectation didn't align with it. In contrast, if an ASCII-holding object fails to be able to contain a semicolon, that's beyond unexpected—that's a bug.
And frankly, I don't think there is a clear expectation about whether things like substr and split, which create brand new strings out of pieces of existing strings, should blindly copy the UTF8 flag of their input.
Ideally, "expected" behavior would still be documented. (That statement strays into tautologyland: documenting things is how users know to expect them.) If the behavior is expected but still intentionally undefined, that fact ought to be documented too, so that coders know not to rely on it. The case at hand is neither, which to me suggests a shortfall in the documentation. A sentence or two in perlunicode about what's not guaranteed regarding the UTF8 flag would solve this, and let coders ensure their code isn't making unwarranted assumptions.
| [reply]
[d/l]
[select] |