I should mention that if you open the input file like this: 
open( $fh, "<:encoding(UTF-16)", $filename );

[download]
(that is, without the "LE" in the encoding spec), then you won't need this line: 
s/\x{feff}//;

[download]
because the "unmarked" version of UTF-16 encoding requires that a stream-initial BOM be provided on input, and the initial BOM is stripped from input as a result.

For output of UTF-16, if you're trying to match a particular byte order, it'll be best for the code to state this explicitly, because the "default" output order might be different, depending on your machine and environment.

Of course, whenever a file is written with 'UTF-16' encoding, the initial BOM is always included, which should make it possible for any other process to read the file correctly - but of course, not all processes that expect UTF-16LE (or BE) will live up to that specification.

Anyway, when you do decide to be explicit about byte order for an output file, then you should also be sure to include the initial BOM (because it won't be supplied by default). So if you try out the snippet below, see whether there's any difference in the output when you comment out the "UTF-16" open statement and uncomment the two lines that use "UTF-16LE" instead: 

open( I, "<:encoding(UTF-16)", $ARGV[0] ) or die "$ARGV[0]: $!";
local $/;
$_=<I>;
@lines = split /\r\n/;

# open(O,">:encoding(UTF-16LE):crlf","$ARGV[0].new") or die "$ARGV[0].
+new:$!";
# print O "\x{feff}";

open(O,">:encoding(UTF-16):crlf","$ARGV[0].new") or die "$ARGV[0].new:
+$!";

print O "$_\n" for ( @lines );

[download]

In reply to Re^3: Problems parsing UTF16 file by graff
in thread Problems parsing UTF16 file by stu23

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.