Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

It likely takes a new Perl programmer a long time to learn how to read and write a Unicode UTF-16 file. The following script would not be obvious to someone just beginning to learn Perl.

#!perl use strict; use warnings; open my $input_fh, '<:encoding(UTF-16LE)', 'Input.txt'; open my $output_fh, '>:encoding(UTF-16LE)', 'Output.txt'; while (my $line = <$input_fh>) { chomp $line; print $output_fh "$line\n"; }

Imagine how glorious the neophyte would feel once he had finally figured out the right way to handle Unicode text in Perl after having slogged through the nearly impenetrable Perl Unicode documentation for hours. Then imagine how frustrating it would be for him to run the script and realize it doesn't work. It creates a badly broken and unusable text file (on Microsoft Windows, at least).

But our neophyte is patient and persistent. He Googles for help, and after several more hours of painstaking research and experimentation, he determines the following script works.

#!perl use strict; use warnings; open my $input_fh, '<:raw:perlio:encoding(UTF-16LE):crlf', 'Input.txt'; open my $output_fh, '>:raw:perlio:encoding(UTF-16LE):crlf', 'Output.txt'; while (my $line = <$input_fh>) { chomp $line; print $output_fh "$line\n"; }

The chicanery needed just to read and write a Unicode file on Windows using Perl is absurd. It's much too arcane.

Can someone explain how this sequence of PerlIO layers works? Why must so many layers be used? Can these layers be specified using the open pragma? If so, how? If not, why not? And why has this ancient Perl bug still not been fixed in version 5.12.2?

In fact, the second, more elaborate version of the script is still wrong. The file named Input.txt has a byte order mark in it, so its encoding is actually UTF-16, not UTF-16LE. It seems there's no way to generate a UTF-16 file in little-endian byte order directly. To generate such a file, you have to specify the UTF-16LE CES (which is wrong) and add the byte order mark explictly to make it UTF-16 instead of UTF-16LE.

#!perl use strict; use warnings; use charnames qw( :full ); open my $input_fh, '<:raw:perlio:encoding(UTF-16):crlf', 'Input.txt'; open my $output_fh, '>:raw:perlio:encoding(UTF-16LE):crlf', 'Output.txt'; print $output_fh "\N{BYTE ORDER MARK}"; while (my $line = <$input_fh>) { chomp $line; print $output_fh "$line\n"; }

In reply to Chicanery Needed to Handle Unicode Text on Microsoft Windows by Jim

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others wandering the Monastery: (3)
    As of 2021-04-23 03:28 GMT
    Find Nodes?
      Voting Booth?

      No recent polls found