Re: File::Find::Rule returns different filenames if they have chars with accents: OSX vs Linux

It's not necessarily File::Find::Rule that arbitrarily changes the representation of the file name.

I recall that a few years ago I had to test the compatibility of our application at $work with various operating systems, and there were differences related to unicode normalization: browsers on OSX tended to return accented characters typed into a password field in their decomposed form (basic letter + combining accent), while those on Windows and Linux returned the composed form (accented letter). Perhaps this reflects a widespread custom on these operating systems, or a feature in an underlying library.

You could try ls | xxd to check the actual representation of your file names in the file system. On my linux box most files (that have accented characters in their names) are in the composed form, but I've found a few that aren't.

Comment on Re: File::Find::Rule returns different filenames if they have chars with accents: OSX vs Linux Download Code

Replies are listed 'Best First'.
Re^2: File::Find::Rule returns different filenames if they have chars with accents: OSX vs Linux by bliako (Abbot) on Jan 02, 2024 at 19:40 UTC
thanks, I have just discovered that now when I decided to test how it fares on a unicoded dir, taking the tarball distribution from OSX (with said dir) to Linux broke the tests while the dirs seemed the same, albeit normalised and unormalised. I will experiment with making the dirs and files during testing with normalised form and see if that is respected by the two OS'es. And I haven't touched windows yet :(( yikes.	[reply]
Re^2: File::Find::Rule returns different filenames if they have chars with accents: OSX vs Linux by bliako (Abbot) on Jan 03, 2024 at 12:23 UTC
I have searched on normalisation and OSX FS and it seems it is a fine mess. See e.g. https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/ . Oh well the unicode curse got apple who thought they could tame it. Apropos: I learned a new [sic] neologism ... `dystextia`.	[reply] [d/l]
Re^3: File::Find::Rule returns different filenames if they have chars with accents: OSX vs Linux by kikuchiyo (Hermit) on Jan 03, 2024 at 14:48 UTC
A fine mess indeed. I've experimented as well, and on linux it's possible to have two (or more) different files whose names appear to be the same because of different unicode normalization. `perl -e 'for (["a\x{0301}", "decomposed"], ["\xc3\xa1", "composed"]) {open my $F, ">", $_->[0]; print $F $_->[1]; close $F}'` After running this: `$ ls -rw-r--r--. 1 user user 8 Jan 3 15:31 á -rw-r--r--. 1 user user 10 Jan 3 15:31 á` [download] (in the terminal these appear identical) If I turn on my native keyboard layout and type "less á", I get the composed file. But I can copy and paste the decomposed string into the terminal, and access the other file as well. Apparently, this filesystem, and linux filesystems in general, don't assume and enforce much about file name encoding: file names are just a sequence of bytes, and the user can keep the pieces. My gut feeling is that Apple's way, that is, normalization, is "better" from an usability standpoint - but then it had better be completely consistent and enforced everywhere.	[reply] [d/l] [select]