in reply to Re^7: Converting Unicode
in thread Converting Unicode

So actually I had forgotten about it, and then when you reminded me I was going to say "oh right, but it doesn't apply to modules, only the main script", and then I gave it a test and actually it works..?

perl -C -E 'use Path::Tiny; say length((path("unicode.txt")->lines)[0])'

but that seems to disagree with the documentation:

The io options mean that any subsequent open() (or similar I/O operations) in main program scope will have the :utf8 PerlIO layer implicitly applied to them, in other words, UTF-8 is expected from any input stream, and UTF-8 is produced to any output stream. This is just the default set via ${^OPEN}, with explicit layers in open() and with binmode() one can manipulate streams as usual. This has no effect on code run in modules.

Though, still, the one thing missing is unicode handling of file names, such as the return values of readdir.

Edit:

So actually the documentation is correct, it only applies to the main module. Path::Tiny is just very smart about doing what you mean, because it calls 

    my $binmode = $args->{binmode};
    $binmode = ( ( caller(0) )[10] || {} )->{'open<'} unless defined $
+binmode;
    my $fh = $self->filehandle( { locked => 1 }, "<", $binmode );

[download]

So, no, -C isn't what I'm talking about. I mean a perl-wide change of defaults that makes all text (non-binmode("raw")) default to decoding UTF-8 in all modules everywhere. ...because that would fix Polyglot's Test::More problem without monkeying around with file handles private within other modules.

Replies are listed 'Best First'.
Re^9: Converting Unicode
by LanX (Saint) on Dec 05, 2023 at 15:03 UTC
    > the one thing missing is unicode handling of file names, such as the return values of readdir

    I don't know much about this and how filesystems are handling encodings.

    I was kind of expecting that this is just another matter of IO layers... (?)

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    see Wikisyntax for the Monastery

[reply]

      In Unix, the filesystem is always just bytes, but all popular modern software is (depending on the LC environment vars) assuming those bytes can be decoded as UTF-8 and does so. In Windows, all paths are unicode, but use an 8-bit locale unless you use 16-bit wide character APIs, and Perl has always been fairly broken when using international filenames on Windows because perl uses the 8-bit APIs. It's only recently that Win10 introduced the UTF-8 Application Codepage that lets Perl see UTF-8 via those 8-bit APIs.

      To the best of my knowledge, Perl only ever sees filenames as bytes and the user must handle all decoding and encoding. It results in a lot of ugly code. I wrote a whole investigative meditation about it, and looked at Python's handling of the problem for comparison. I also suggested solving it as part of a virtual filesystem module for perl.

      Meanwhile, I'm a native English speaker and the only time I run into these problems are when filenames of my music collection use foreign characters, or a few cases where I was trying to make backups of client files that contain smart quotes. I can only imagine how frustrating this would be to someone with an asian language who probably uses UTF-8 for every directory and filename. Python 3 has "solved" the problem about as much as it can be solved, and I wouldn't expect to get many new perl users from asian countries if this is one of the problems they run into regularly. Or in other words, I think it ought to be a higher priority to fix this.

[reply]
        Thank you so much for chiming in here. Perl is not fully unicode compatible yet...but people who don't use unicode regularly, particularly Asian scripts, will likely be oblivious to this and unable to understand the situation. Your points are valid and need more attention.

        I attended a week of Python training last year. At the time I smugly felt Perl to be superior in many ways. Now I'm wondering if I should pursue it more seriously. Python does have its advantages, even if I feel bothered by its strict formatting rules.

        Blessings,

        ~Polyglot~

[reply]

In Section Seekers of Perl Wisdom