Learned the hard way that NTFS would allow file names to be written to FAT32 even if some characters are outside of FAT32 specification.

NTFS won't write to FAT32. NTFS is a filesystem just like FAT32. And so the NTFS driver will only ever write to NTFS-formatted media, as will the (V)FAT driver only ever write to (V)FAT-formatted media.

FAT32 isn't the problem here. Windows' history is the problem explaining both the "forbidden" names (AUX, CON, PRN, NUL, ...) and the "forbidden" characters.

The first problem is due to someone during the early history of MS-DOS deciding that having a separate \DEV\ directory for Unix-style devices was not needed, and so the devices just existed in every directory, invisible and not in the filesystem, but still kind-of there. Early versions of MS-DOS still accepted the \DEV\ directory, even if it did not exist. This grew to bug-compatibility. In DOS batch files, you could use IF EXIST SOMENAME to check if a file named SOMENAME existed. But EXIST did not find directories, only files. To check if a directory existed, you had to use a bug, where EXIST finds all of the "forbidden" device names, because they kind-of exist in every existing directory. So, you use IF EXIST SOMENAME\NUL to check if a directory named SOMENAME exists.

The second problem is that ancient MS-DOS with its strictly limited filenames mostly did get the job done without proper quoting rules. Yes, it was not possible to pass <, >, and a few other characters as arguments to a program, but it was just a little bit annoying. "Long filenames" (VFAT) came and removed a lot of filename restrictions, quotes were needed, but nobody made a clear set of rules. It was a mess, and became worse and worse, as quoting and unquoting needs to be implemented by the applications and it is handled differently by different implementations. See Re^3: Having to manually escape quote character in args to "system"? for some ugly details.

If you do not need to interact with DOS/Windows, e.g. because your firmware expects a FAT filesystem for booting (UEFI-based PCs, Raspberry Pi, embedded systems, ...), you are free to use "forbidden" names and characters on a FAT filesystem. You could even make the filesystem case-sensitive or store long ("VFAT") filenames as UTF-16 instead of UCS-2. The firmware might expect some DOS compatibility, so you will probably not be able to boot your system from a file named "\" or "<". But perhaps it will boot from "AUX".

Alexander