http://qs321.pair.com?node_id=687481


in reply to Module to read - modify - write text files in any unicode encoding

While my File::AnyEncoding::write_file for UTF-16LE test file correctly converts "\n" into 0D 00 0A 00, my File::AnyEncoding::read_file for UTF-16LE does not convert this back to "\n" but to "\r\n", why ?

I haven't (yet) looked at your code in detail, but my suspicion is that your problem has to do with a bug in the crlf PerlIO layer, which is that it doesn't correctly handle the multibyte encodings UTF-16, UTF-32, UCS-2 and UCS-4. I.e., it does its translation magic only on the byte sequences 0D 0A (on input) or 0A (on output).

In other words, you'd have to rearrange the PerlIO layer stack in such a way that the crlf layer happens to be applied to an encoding (e.g. UTF-8) where the bytes 0D and 0A appear next to each other in the stream (no 00 in between) — like you already seem to be trying in your write_file() routine.  Also see PerlIO: crlf layer on Windows interfering with UCS-2 unicode.

Update: Maybe it's worth pointing out that you want the kludge (i.e. the layer stack ":raw:encoding($enc):crlf:utf8") only on Windows, where \r\n <—> \n translation is supposed to happen, and only if the encoding in question is representing \r and \n as more than one byte (or more precisely, not exactly as 0D and 0A, i.e. for example UTF-16, UTF-32, UCS-2, UCS-4 — note that UTF-8 and most other encodings are fine). In all other cases, you should use the normal ":encoding($enc)" approach.