Re: Converting UTF-16 files to UTF-8

In Windows world the default "Unicode" encoding is UTF-16 little endian, which I'm guessing is what UltraEdit saves. However, in Perl's Encode world, "utf16" without a byte-order specified assumes big-endian by default so you need to say "utf16le" if you mean the other thing.

That said, it seems to me the presence of the BOM (which is mandatory) should mean, at least theoretically, that Perl could figure out for itself which one it is and it wouldn't go all kablooey. But I could be wrong.

Update: It occurs to me that I may have this entirely backwards. But I think the general idea is at least close - that is, you're saving the file in one byte order and Perl's reading it in another.

Comment on Re: Converting UTF-16 files to UTF-8

Replies are listed 'Best First'.
Re^2: Converting UTF-16 files to UTF-8 by graff (Chancellor) on May 17, 2007 at 13:44 UTC
I don't know where this is documented (I'm sure it must be somewhere), but having played with it a bit, it seems that perl's "UTF-16" (no "BE" or "LE") means "There needs to be a BOM at the start". Perl will write a BOM on initial output to a file handle that is set for this encoding (and will use your machine's native byte order). On initial input, it will error out with "UTF-16:Unrecognized BOM xxxx" unless the first two bytes are either 0xFF 0xFE or 0xFE 0xFF. (And yes, if the first two bytes are one of those two pairings, it will use the given byte order.)	[reply]

Replies are listed 'Best First'.

Re^2: Converting UTF-16 files to UTF-8
by graff (Chancellor) on May 17, 2007 at 13:44 UTC

Perl will write a BOM on initial output to a file handle that is set for this encoding (and will use your machine's native byte order). On initial input, it will error out with "UTF-16:Unrecognized BOM xxxx" unless the first two bytes are either 0xFF 0xFE or 0xFE 0xFF. (And yes, if the first two bytes are one of those two pairings, it will use the given byte order.)

[reply]