comment on

I think it would be very nice to have a facility to know how to encode filesystem strings, and a library that performs that job so that doing it right is easier than not

The problem is that at least some filesystems just don't care about encoding:

NTFS is easy, it uses UTF-16 to encode names, and forbits only a few codepoints depending on the operating system (POSIX forbits / and NUL, as usual. Windows forbids /\:*"?<>| and NUL.)
The Apple File System, like HFS Plus, found on OS X Macintosh systems is encoded in UTF-8.
FAT uses an 8-bit-encoding depending on the operating system, CP437 on old DOS machines, CP850 on newer DOS machines in the "western" parts of the world, or some completely different encoding in other parts of the world and on non-DOS-Machines (e.g. Atari). FAT does NOT store any information about which encoding is used. Excluded characters depend on the OS, additionally, character 0xE5 is used to mark deleted files. On DOS machines, names are converted to upper case. Yes, FAT smells funny, but because it is used for the EFI system partition, it won't go away any time soon.
FAT extended with Long filename support à la Microsoft (often called "VFAT") uses UCS-2 for the "long" filenames.
exFAT (which does not look like classic FAT at all) uses UTF-16 to encode filenames. Microsoft forbids U+0000 to U+001F, and /\:*?"<>| in filenames.
ext2, ext3, and ext4, like FAT, use an 8-bit encoding depending on the operating system, and forbid only NUL and / in filenames. Many Linux distributions assume that filenames are encoded in UTF-8, but that's completely in user space, not in the filesystem.
The same is true for btrfs.
And for xfs.
And for zfs.
And for ufs.
And for ReiserFS.

There are many more filesystems, but I think these are still commonly in use on personal computers and PC-based servers.

As you can see, only a few filesystems use some variant of Unicode encoding for the filenames. The others use just bytes, with some "forbidden" values (eFAT), in some encoding that can not be derived from the filesystem. Userspace may decide to use UTF-8 or some legacy encoding on those filesystems.

As long as we stay in user space (as opposed to kernel space), we don't have to care much about the encoding. Converting the encoding is the job of the operating system. Legacy filesystems have their encoding set via mount options or the like.

Systems like Linux and the *BSDs just don't care about encoding, they treat filenames as null-terminated collections of bytes, just like Unix did since 1970. Modern Windows can treat filenames as Unicode when using the "wide API" (function names ending in a capital W) where all strings are using UCS-2 or UTF-16. When using the "ASCII API" (function names ending in a capital A), strings are using some legacy 8-bit encoding based on ASCII, the actual encoding used depends on user and regional settings. Plan 9 from Bell Labs used UTF-8 for the entire API. (I don't know how Apple handles filenames in their APIs.)

So, how should your proposed library handle file names?

On Windows, it should use the "wide API", period. Everything is Unicode (UTF-16/UCS-2). Same for Plan 9, but using UTF-8 encoding.

Linux and the BSDs? Everything is a null-terminated collection of bytes, maybe UTF-8, maybe some legacy encoding, and maybe also depending on where in the filesystem you are. How should your library handle that?

Oh, and let's not forget that Android is technically Linux, with a lot of custom user-space on top.

Mac OS X? I don't know the APIs, but it inherited a lot from Unix. So it propably looks like Unix.

I completely omitted other command line parameters, the environment, and I/O via STDIN/STDOUT/STDERR. Command line and environment are more or less just an API thing, like filenames. I/O is completely different. It lacks any information about the encoding and is treated as a byte stream on Unix. Windows treats it as a character stream with an unspecified 8-bit encoding, converting line endings as needed, but not characters.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

In reply to Re^6: how are ARGV and filename strings represented? by afoken
in thread how are ARGV and filename strings represented? by almr

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.