While my File::AnyEncoding::write_file for UTF-16LE test file
correctly converts "\n" into 0D 00 0A 00, my File::AnyEncoding::read_file
for UTF-16LE does not convert this back to "\n" but to "\r\n", why ?
I haven't (yet) looked at your code in detail, but my suspicion is
that your problem has to do with a bug in the crlf PerlIO layer, which
is that it doesn't correctly handle the multibyte encodings UTF-16, UTF-32, UCS-2 and UCS-4.
I.e., it does its translation magic only on the byte sequences 0D 0A (on
input) or 0A (on output). In other words, you'd have to rearrange the
PerlIO layer stack in such a way that the crlf layer happens to
be applied to an encoding (e.g. UTF-8) where the bytes 0D and 0A appear
next to each other in the stream (no 00 in between) — like you already seem to be trying in your write_file() routine.
Also see PerlIO: crlf layer on Windows interfering with UCS-2 unicode.
Update: Maybe it's worth pointing out that you want the
kludge (i.e. the layer stack ":raw:encoding($enc):crlf:utf8")
only on Windows, where \r\n <—> \n translation is supposed
to happen, and only if the encoding in question is representing \r and
\n as more than one byte (or more precisely, not exactly as 0D and 0A, i.e. for example UTF-16, UTF-32, UCS-2, UCS-4 —
note that UTF-8 and most other encodings are fine). In all other cases,
you should use the normal ":encoding($enc)" approach.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|