in reply to Re^9: Converting Unicode
in thread Converting Unicode
In Unix, the filesystem is always just bytes, but all popular modern software is (depending on the LC environment vars) assuming those bytes can be decoded as UTF-8 and does so. In Windows, all paths are unicode, but use an 8-bit locale unless you use 16-bit wide character APIs, and Perl has always been fairly broken when using international filenames on Windows because perl uses the 8-bit APIs. It's only recently that Win10 introduced the UTF-8 Application Codepage that lets Perl see UTF-8 via those 8-bit APIs.
To the best of my knowledge, Perl only ever sees filenames as bytes and the user must handle all decoding and encoding. It results in a lot of ugly code. I wrote a whole investigative meditation about it, and looked at Python's handling of the problem for comparison. I also suggested solving it as part of a virtual filesystem module for perl.
Meanwhile, I'm a native English speaker and the only time I run into these problems are when filenames of my music collection use foreign characters, or a few cases where I was trying to make backups of client files that contain smart quotes. I can only imagine how frustrating this would be to someone with an asian language who probably uses UTF-8 for every directory and filename. Python 3 has "solved" the problem about as much as it can be solved, and I wouldn't expect to get many new perl users from asian countries if this is one of the problems they run into regularly. Or in other words, I think it ought to be a higher priority to fix this.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^11: Converting Unicode
by Polyglot (Chaplain) on Dec 06, 2023 at 02:27 UTC | |
by jeffenstein (Hermit) on Dec 06, 2023 at 09:28 UTC | |
by Polyglot (Chaplain) on Dec 06, 2023 at 10:29 UTC | |
by jeffenstein (Hermit) on Dec 06, 2023 at 14:47 UTC |