in reply to substr on UTF-8 strings

The Perl string with the file path was originally read from a UTF-8 text file and is internally flagged as UTF-8.

So you say you have decoded text (aka a string of Unicode Code Points) which is stored using the UTF=8 internal storage format?

However, the string I get back is always flagged as "native/raw bytes".

Perl is free to pick whatever internal storage format it wants.

That said, I can't reproduce your claim. substr returns a string using the UTF8=1 format if that's the storage format used by the input string. 

$ perl -e'
   use Devel::Peek qw( Dump );
   my $s = "a\N{U+2660}";
   Dump($s);
   my $ss = substr($s, 0, 1);
   Dump($ss);
'
SV = PV(0x7fffd3496ca0) at 0x7fffd34c5a88
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK,UTF8)
  PV = 0x7fffd34c33a0 "a\342\231\240"\0 [UTF8 "a\x{2660}"]
  CUR = 4
  LEN = 10
  COW_REFCNT = 1
SV = PV(0x7fffd3496d30) at 0x7fffd34c5ad0
  REFCNT = 1
  FLAGS = (POK,pPOK,UTF8)
  PV = 0x7fffd34ad050 "a"\0 [UTF8 "a"]
  CUR = 1
  LEN = 10

[download]

Replies are listed 'Best First'.
Re^2: substr on UTF-8 strings
by ikegami (Patriarch) on Jun 26, 2020 at 19:42 UTC

    Ah, if the input string contains only characters in 00..7F, the returned string will use the UTF8=0 internal storage format. 

    $ perl -e'
   use Devel::Peek qw( Dumpe );
   my $s = "ab";
   utf8::upgrade($s);  # Force UTF8=1 storage format.
   Dump($s);
   my $ss = substr($s, 0, 1);
   Dump($ss);
'
SV = PV(0x7fffdcf3dca0) at 0x7fffdcf6ca78
  REFCNT = 1
  FLAGS = (POK,pPOK,UTF8)
  PV = 0x7fffdcf5dbf0 "ab"\0 [UTF8 "ab"]
  CUR = 2
  LEN = 10
SV = PV(0x7fffdcf3dd30) at 0x7fffdcf6cb50
  REFCNT = 1
  FLAGS = (POK,pPOK)
  PV = 0x7fffdcf54050 "a"\0
  CUR = 1
  LEN = 10

    [download]

    But like I said, it's Perl's perogative to pick whatever internal storage format it wants.

[reply]
[d/l]

In Section Seekers of Perl Wisdom