Re: Chicanery Needed to Handle Unicode Text on Microsoft Windows

in reply to Chicanery Needed to Handle Unicode Text on Microsoft Windows

As far as I can see, the only reason you need :crlf is because you've specifically added the UNIX line ending (\n) to your output. It would be better to use the platform-independent $/. The :raw layer should preserve the line endings. So that reduces the chicanery somewhat.

Except for ASCII files, binmode($file_handle) was required on MSWin32 systems. :raw performs the same function so, while perhaps appearing to add to the chicanery, it certainly reduces the amount of code.

I don't have sufficient knowledge of UTF-16 to address that aspect of you post. What I would suggest is that, after removing :crlf and changing \n to $/, you try your test code without :perlio. You may still need it but it wouldn't hurt to check.

I agree there's a lot of Unicode-related documentation; however, everything I've made reference to is available here: PerlIO.

I ran a series of tests, click on Read more... to view.

Starting code:

#!perl

use 5.12.0;
use warnings;

my $in_file = $^O eq 'MSWin32'
            ? 'utf16_LE_prob.dos_dat'
            : 'utf16_LE_prob.unix_dat';

my $out_file = $^O eq 'MSWin32'
            ? 'utf16_LE_prob.dos_out'
            : 'utf16_LE_prob.unix_out';

my $in_mode = $^O eq 'MSWin32'
            ? '<:raw'
            : '<';

my $out_mode = $^O eq 'MSWin32'
            ? '>:raw'
            : '>';

open my $in_fh, $in_mode, $in_file or die $!;
open my $out_fh, $out_mode, $out_file or die $!;

while (my $line = <$in_fh>) {
    print $out_fh $line;
}

close $out_fh;
close $in_fh;
[download]

Input files in UNIX and DOS formats:

$ cat -vet utf16_LE_prob.unix_dat utf16_LE_prob.dos_dat
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

Output after running on UNIX platform:

$ cat -vet utf16_LE_prob.unix_out
Line 1$
Line 2$
$
[download]

Output after running on DOS platform:

$ cat -vet utf16_LE_prob.dos_out
Line 1^M$
Line 2^M$
^M$
[download]

Changing the while loop to chomp input and add $/ (not \n) to output:

while (my $line = <$in_fh>) {
    chomp $line,
    print $out_fh $line, $/;
}
[download]

New output:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

Adding :crlf to MSWin32 input and output modes (now = :raw:crlf) and there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

With :raw:perlio:crlf, there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

And, for completeness, with :raw:perlio, there's no change:

$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out
Line 1$
Line 2$
$
Line 1^M$
Line 2^M$
^M$
[download]

-- Ken

In Section Seekers of Perl Wisdom