in reply to Chicanery Needed to Handle Unicode Text on Microsoft Windows
As far as I can see, the only reason you need :crlf is because you've specifically added the UNIX line ending (\n) to your output. It would be better to use the platform-independent $/. The :raw layer should preserve the line endings. So that reduces the chicanery somewhat.
Except for ASCII files, binmode($file_handle) was required on MSWin32 systems. :raw performs the same function so, while perhaps appearing to add to the chicanery, it certainly reduces the amount of code.
I don't have sufficient knowledge of UTF-16 to address that aspect of you post. What I would suggest is that, after removing :crlf and changing \n to $/, you try your test code without :perlio. You may still need it but it wouldn't hurt to check.
I agree there's a lot of Unicode-related documentation; however, everything I've made reference to is available here: PerlIO.
I ran a series of tests, click on Read more... to view.
Starting code:
#!perl use 5.12.0; use warnings; my $in_file = $^O eq 'MSWin32' ? 'utf16_LE_prob.dos_dat' : 'utf16_LE_prob.unix_dat'; my $out_file = $^O eq 'MSWin32' ? 'utf16_LE_prob.dos_out' : 'utf16_LE_prob.unix_out'; my $in_mode = $^O eq 'MSWin32' ? '<:raw' : '<'; my $out_mode = $^O eq 'MSWin32' ? '>:raw' : '>'; open my $in_fh, $in_mode, $in_file or die $!; open my $out_fh, $out_mode, $out_file or die $!; while (my $line = <$in_fh>) { print $out_fh $line; } close $out_fh; close $in_fh;
Input files in UNIX and DOS formats:
$ cat -vet utf16_LE_prob.unix_dat utf16_LE_prob.dos_dat Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$
Output after running on UNIX platform:
$ cat -vet utf16_LE_prob.unix_out Line 1$ Line 2$ $
Output after running on DOS platform:
$ cat -vet utf16_LE_prob.dos_out Line 1^M$ Line 2^M$ ^M$
Changing the while loop to chomp input and add $/ (not \n) to output:
while (my $line = <$in_fh>) { chomp $line, print $out_fh $line, $/; }
New output:
$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$
Adding :crlf to MSWin32 input and output modes (now = :raw:crlf) and there's no change:
$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$
With :raw:perlio:crlf, there's no change:
$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$
And, for completeness, with :raw:perlio, there's no change:
$ cat -vet utf16_LE_prob.unix_out utf16_LE_prob.dos_out Line 1$ Line 2$ $ Line 1^M$ Line 2^M$ ^M$
-- Ken
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^2: Chicanery Needed to Handle Unicode Text on Microsoft Windows
by Jim (Curate) on Oct 30, 2010 at 18:36 UTC | |
by kcott (Archbishop) on Oct 31, 2010 at 02:07 UTC | |
Re^2: Chicanery Needed to Handle Unicode Text on Microsoft Windows
by Anonymous Monk on Oct 30, 2010 at 10:47 UTC | |
by kcott (Archbishop) on Oct 31, 2010 at 02:09 UTC |