comment on

As far as I can see, the only reason you need :crlf is because you've specifically added the UNIX line ending (\n) to your output. It would be better to use the platform-independent $/. The :raw layer should preserve the line endings. So that reduces the chicanery somewhat.

Except for ASCII files, binmode($file_handle) was required on MSWin32 systems. :raw performs the same function so, while perhaps appearing to add to the chicanery, it certainly reduces the amount of code.

I don't have sufficient knowledge of UTF-16 to address that aspect of you post. What I would suggest is that, after removing :crlf and changing \n to $/, you try your test code without :perlio. You may still need it but it wouldn't hurt to check.

I agree there's a lot of Unicode-related documentation; however, everything I've made reference to is available here: PerlIO.

I ran a series of tests, click on Read more... to view.

Starting code:

#!perl

use 5.12.0;
use warnings;

my $in_file = $^O eq 'MSWin32'
            ? 'utf16_LE_prob.dos_dat'
            : 'utf16_LE_prob.unix_dat';

my $out_file = $^O eq 'MSWin32'
            ? 'utf16_LE_prob.dos_out'
            : 'utf16_LE_prob.unix_out';

my $in_mode = $^O eq 'MSWin32'
            ? '<:raw'
            : '<';

my $out_mode = $^O eq 'MSWin32'
            ? '>:raw'
            : '>';

open my $in_fh, $in_mode, $in_file or die $!;
open my $out_fh, $out_mode, $out_file or die $!;

while (my $line = <$in_fh>) {
    print $out_fh $line;
}

close $out_fh;
close $in_fh;
[download]

Input files in UNIX and DOS formats:

$ cat -vet utf16_LE_prob.unix_dat utf16_LE_prob.dos_dat
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

Output after running on UNIX platform:

$ cat -vet utf16_LE_prob.unix_out
Line 1$
Line 2$
$
[download]

Output after running on DOS platform:

$ cat -vet utf16_LE_prob.dos_out
Line 1^M$
Line 2^M$
^M$
[download]

Changing the while loop to chomp input and add $/ (not \n) to output:

while (my $line = <$in_fh>) {
    chomp $line,
    print $out_fh $line, $/;
}
[download]

New output:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

Adding :crlf to MSWin32 input and output modes (now = :raw:crlf) and there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

With :raw:perlio:crlf, there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

And, for completeness, with :raw:perlio, there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

-- Ken

In reply to Re: Chicanery Needed to Handle Unicode Text on Microsoft Windows by kcott
in thread Chicanery Needed to Handle Unicode Text on Microsoft Windows by Jim

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Come for the quick hacks, stay for the epiphanies.
	PerlMonks