in reply to how to determine the text file encoding
(I'm guessing) Perl won't output a BOM character unless you tell it to do so using print(chr(0xFEFF));. While I can't run your program with my version of Perl, the print I mentioned gives me EF BB BF as expected. It makes sense for Perl to not output the BOM automatically because it is not always necessary, and sometimes it isn't even permitted. According to unicode.org,
Where the data is typed, such as a field in a database, a BOM is unnecessary. In particular, if a text data stream is marked as UTF-16BE, UTF-16LE, UTF-32BE or UTF-32LE, a BOM is neither necessary nor permitted. Any FEFF would be interpreted as a ZWNBSP.
|
|---|