Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: RFC: system calls on Unicode filesystem

by salva (Canon)
on Feb 28, 2018 at 08:06 UTC ( [id://1210063]=note: print w/replies, xml ) Need Help??


in reply to RFC: system calls on Unicode filesystem

The broken handling of Unicode on file system operations is not just a Windows issue. It is not done correctly in Unix/Linux either:

See how the UTF8 flag is completely ignored in scalars passed as arguments to built-ins performing file system operations in the following re.pl session:

salva@atun:/tmp/unicode$ re.pl $ $a="a\xf1o" a�o $ $b = $a a�o $ use Devel::Peek $ utf8::upgrade($b) 4 $ Dump $a SV = PV(0x557576d4e2d0) at 0x557576812a80 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x557576d516c0 "a\361o"\0 CUR = 3 LEN = 10 COW_REFCNT = 0 $ Dump $b SV = PV(0x557576d4e2a0) at 0x557576d4b7d0 REFCNT = 1 FLAGS = (POK,pPOK,UTF8) PV = 0x557576d6e6e0 "a\303\261o"\0 [UTF8 "a\x{f1}o"] CUR = 4 LEN = 10 $ open A, ">$a"; 1 $ open B, ">$b"; 1 $ system "ls" año a?o 0 $ $a eq $b 1

Replies are listed 'Best First'.
Re^2: RFC: system calls on Unicode filesystem
by daxim (Curate) on Feb 28, 2018 at 10:21 UTC
    This is wrong, it has nothing to do with the UTF8 flag, see demo program below. The reason why open A in your example creates a file with a name that is broken for display purposes only¹ is because you (implicitly) used an encoding (namely, Latin1) that does not agree with the encoding of the filesystem² (namely, UTF-8).

    If a programmer does the straight-forward thing that comes to mind, namely just using a string, not even bothering to encode to octets, then there is no problem. Both strings and UTF-8 encoded octet sequences work. I personally haven't even run into the problem during the last ten years because it requires special action to set oneself up for failure in order to trigger.

    To come back to the topic expressed in the root post, I think nothing needs to be done for Posix. This is a Windows only bugfix.

    ¹ The big difference to the Windows shitshow where system calls do not work and return error messages, on Posix they do work even if you somehow fuck up the encoding. The file will exist, will be enumerable, will create a filehandle you can operate on, merely the name will display wrong.

    ² De jure the encoding of a Posix filesystem is unknown and unknowable, but de facto it is UTF-8. I remember dwheeler is doing work to get UTF-8 as default into the next version of the standard, but I can't find the relevant document on the Web.

    use 5.026;
    use utf8;
    use Devel::Peek qw(Dump);
    
    my $string = 'string-año';
    #   FLAGS = (POK,IsCOW,pPOK,UTF8)
    #   PV = 0x12f4760 "string-a\303\261o"\0 [UTF8 "string-a\x{f1}o"]
    my $octets_latin1 = "octets-latin1-a\x{f1}o";
    #   FLAGS = (POK,IsCOW,pPOK)
    #   PV = 0x12f4420 "octets-latin1-a\361o"\0
    my $octets_utf8 = "octets-utf8-a\x{c3}\x{b1}o";
    #   FLAGS = (POK,IsCOW,pPOK)
    #   PV = 0x12ebb00 "octets-utf8-a\303\261o"\0
    
    my $fh;                         # ↓ displays in shell as: ↓
    open $fh, '>', $string;         # string-año
    open $fh, '>', $octets_latin1;  # 'octets-latin1-a'$'\361''o'
    open $fh, '>', $octets_utf8;    # octets-utf8-año
    
      My point is that two variables that eq sees as containing the same value, create two different files just because of its internal representation.

      The representation used by perl internally, should not affect external interactions.

      Update: BTW, note that in my sample repl above I have used utf8::upgrade which just changes the representation. Quoting utf8:

      Converts in-place the internal representation of the string from an octet sequence in the native encoding (Latin-1 or EBCDIC) to UTF-8.The logical character sequence itself is unchanged.
        I have written three attempts at replies with technical details and but I decided to delete them again. I understand all nuances of the problem fully and I am certain you are, too, so writing out the technical details is pointless. We are both looking at a cone and one from the side sees a triangle and one from the bottom sees a circle.

        The difference between our perspectives:

        • I chose to live in a world where 'use 5.014; use utf8;' is always in effect.
        • I never use the functions internal to the utf8 module because its documentation says so.
        • It is irrelevant that Perl says $a eq $b is true. In my mind, they are different types. $a is a buf of octets, $b is a string of characters. I have trained myself to notice that it does not even make sense to test for equality without converting one of them first. The only way to convert between the types is the Encode module.
        • Representations are an irrelevant concept. If I have a string, I know it's UTF-8, and system calls on Posix will happily do the right thing.
        • By applying the discipline of the best practices rigorously I will not possibly get myself into a situation where the problem you showed can occur.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1210063]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (8)
As of 2024-04-18 10:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found