In Unix, the filesystem is always just bytes, but all popular modern software is (depending on the LC environment vars) assuming those bytes can be decoded as UTF-8 and does so. In Windows, all paths are unicode, but use an 8-bit locale unless you use 16-bit wide character APIs, and Perl has always been fairly broken when using international filenames on Windows because perl uses the 8-bit APIs. It's only recently that Win10 introduced the UTF-8 Application Codepage that lets Perl see UTF-8 via those 8-bit APIs.
To the best of my knowledge, Perl only ever sees filenames as bytes and the user must handle all decoding and encoding. It results in a lot of ugly code. I wrote a whole investigative meditation about it, and looked at Python's handling of the problem for comparison. I also suggested solving it as part of a virtual filesystem module for perl.
Meanwhile, I'm a native English speaker and the only time I run into these problems are when filenames of my music collection use foreign characters, or a few cases where I was trying to make backups of client files that contain smart quotes. I can only imagine how frustrating this would be to someone with an asian language who probably uses UTF-8 for every directory and filename. Python 3 has "solved" the problem about as much as it can be solved, and I wouldn't expect to get many new perl users from asian countries if this is one of the problems they run into regularly. Or in other words, I think it ought to be a higher priority to fix this.
In reply to Re^10: Converting Unicode
by NERDVANA
in thread Converting Unicode
by BernieC
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |