Of course length can also give you different results if you change the UTF8 flag.

I believe you are thinking in terms of particular encodings, which knowledge of the Unicode bug wrongly tempts one to do. I'm not talking about taking an internal representation as a sequence of bytes and then flipping the UTF8 flag on that internal representation, I'm talking about the actual strings represented by the internal representation.

A string in Perl is a sequence of characters, not the sequence of bytes (or octets) that represents those characters in a particular encoding. If length() gives a different answer on two strings, then they are not the same sequence of characters.

In a Unicode world there is one string "fu\x{df}" consisting of three characters. Internally Perl might encode that in one of two different ways, resulting in different byte sequences and a different setting of the UTF8 flag, but it is the same string whichever encoding is used.

So in the code below, I would expect verify_upgraded_length and verify_downgraded_length to return a TRUE value for every string input (if they return at all).

use utf8 ();
sub verify_upgraded_length {
  my($s) = @_;
  # wrong
  # my $u = utf8::upgrade($s);
  my $u = $s;
  utf8::upgrade($u);
  return length($s) == length($u);
}
sub verify_downgraded_length {
  my($s) = @_;
  # wrong
  # my $d = utf8::downgrade($s);  # dies if downgrade not possible
  my $d = $s;
  utf8::downgrade($d);  # dies if downgrade not possible
  return length($s) == length($d);
}
[download]

[Updated: corrected code, thanks haj++]

Nitpicking (again): utf8::upgrade and utf8::downgrade do not return converted strings. They do the change inplace, and return a success indicator instead.

One solution is to take a copy (also note that for current Perls, use utf8 isn't needed):

sub verify_upgraded_length {
  my($s) = @_;
  my $u = $s;
  utf8::upgrade($u);
  return length($s) == length($u);
}
sub verify_downgraded_length {
  my($s) = @_;
  my $d = $s;
  utf8::downgrade($d);  # dies if downgrade not possible
  return length($s) == length($d);
}
[download]

> A string in Perl is a sequence of characters, not the sequence of bytes (or octets) that represents those characters in a particular encoding

Unfortunately that's not the terminology of Perldocs!

• Text strings, or character strings are made of characters. Bytes are irrelevant here
• Binary strings, or byte strings are made of bytes. Here, you don't have characters, just bytes

The latter (AKA Octet-Streams in other docs) lets lc work on "NON characters" b/c of backwards compatibility to Perl4.

And the defining difference between both types of strings is the UTF8-Flag.

Cheers Rolf
_{(addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
Wikisyntax for the Monastery}

Of course length can also give you different results if you change the UTF8 flag.

Could you please give an example for this? How do you "change the UTF8 flag"?

• utf8::encode and utf8::decode change the flag, but also the characters (same for Encode). So, obviously, the length can change.
• utf8::upgrade and utf8::downgrade change the flag and the internal representation, but keep the length.

> Could you please give an example for this? How do you "change the UTF8 flag"?

use v5.12.0;
use warnings;
#use Devel::Peek;

use utf8;
use Encode qw(is_utf8 _utf8_on _utf8_off);

my $str = "ä";
say $str, ":",length($str);

#Dump($str);

_utf8_off($str);
say $str, ":",length($str);

#Dump($str);
[download]

ä:1
ä:2
[download]

Cheers Rolf
_{(addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
Wikisyntax for the Monastery}