While my File::AnyEncoding::write_file for UTF-16LE test file correctly converts "\n" into 0D 00 0A 00, my File::AnyEncoding::read_file for UTF-16LE does not convert this back to "\n" but to "\r\n", why ?

I haven't (yet) looked at your code in detail, but my suspicion is that your problem has to do with a bug in the crlf PerlIO layer, which is that it doesn't correctly handle the multibyte encodings UTF-16, UTF-32, UCS-2 and UCS-4. I.e., it does its translation magic only on the byte sequences 0D 0A (on input) or 0A (on output).

In other words, you'd have to rearrange the PerlIO layer stack in such a way that the crlf layer happens to be applied to an encoding (e.g. UTF-8) where the bytes 0D and 0A appear next to each other in the stream (no 00 in between) — like you already seem to be trying in your write_file() routine.  Also see PerlIO: crlf layer on Windows interfering with UCS-2 unicode.

Update: Maybe it's worth pointing out that you want the kludge (i.e. the layer stack ":raw:encoding($enc):crlf:utf8") only on Windows, where \r\n <—> \n translation is supposed to happen, and only if the encoding in question is representing \r and \n as more than one byte (or more precisely, not exactly as 0D and 0A, i.e. for example UTF-16, UTF-32, UCS-2, UCS-4 — note that UTF-8 and most other encodings are fine). In all other cases, you should use the normal ":encoding($enc)" approach.

In reply to Re: Module to read - modify - write text files in any unicode encoding by almut
in thread Module to read - modify - write text files in any unicode encoding by Rudif

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.