Re^7: how are ARGV and filename strings represented?

Things get even more interesting if you try to use filenames containing invalid UTF-8 sequences on various filenames:

Invalid-UTF8 vs the filesystem by Kristian Köhntopp. In summary:

XFS on Linux and ext4 on Linux don't care at all. Filenames are just bytes.
ZFS on Linux refuses filenames containing invalid UTF-8 sequences.
APFS on MacOS Ventura also refuses filenames containing invalid UTF-8 sequences.

Python does not like tar archives with invalid UTF-8 sequences.

And little ugly detail: Apparently there is a function sys.getfilesystemencoding() without parameters. Python seems to assume that all filesystems have the same encoding and that it is not path dependent.

This is at least conceptually similar to my pet problem of File::Spec, assuming uniform behaviour across various mounted filesystems.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Comment on Re^7: how are ARGV and filename strings represented?

Replies are listed 'Best First'.
Re^8: how are ARGV and filename strings represented? by jeffenstein (Hermit) on Oct 02, 2024 at 08:10 UTC
Just playing devil's advocate: On linux it looks like it's possible to mount external filesystems (vfat, ntfs, iso9660?) with other encodings. What happens if the path up to the mount point is in utf8 and the path after the mount point is in another encoding? It's true that no sane person would try this, but it is possible...	[reply]
Re^9: how are ARGV and filename strings represented? by afoken (Chancellor) on Oct 02, 2024 at 18:08 UTC
It sure is possible, and it does not sound that insane if you are trying to recover data or the like. Just imagine working with legacy media (retro computing) or from other operating systems (e.g. MacOS, Windows, OS/2, ...). I did not assume that any filesystem would check the filenames for correct UTF-8 encoding, especially not ZFS, which comes from an old Unix and in my mind should store just bytes composing filenames. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]