I have two example files
poc1.pl:
use strict;
use warnings;
use Devel::Peek;
use Encode;
use utf8;
my $string = "123\x{444}\x{444}\x{444}\x{444}";
binmode STDOUT, ":utf8";
Dump $string;
print "UTF IS ON\n" if utf8::is_utf8($string);
print "LENGTH DIFFERS\n" if length($string) != bytes::length($string);
open my $f, ">", "test1";
binmode $f;
syswrite $f, $string or die;
print "ALL OK\n";
__END__
SV = PV(0x258cb78) at 0x25b7bb0
REFCNT = 1
FLAGS = (PADMY,POK,pPOK,UTF8)
PV = 0x25aad60 "123\321\204\321\204\321\204\321\204"\0 [UTF8 "123\x{
+444}\x{444}\x{444}\x{444}"]
CUR = 11
LEN = 16
UTF IS ON
LENGTH DIFFERS
Wide character in syswrite at poc1.pl line 16.
poc2.pl:
use strict;
use warnings;
use Devel::Peek;
use Encode;
use utf8;
my $utfstring = "123 \x{439}\x{439}\x{439}\x{439}";
my ($ascii_but_utf, undef) = split ' ', $utfstring;
my $bytestring = encode ("UTF-8", "\x{444}\x{444}\x{444}\x{444}");
my $mixedstring = "$ascii_but_utf$bytestring"; # simulate The Unicode
+Bug here
binmode STDOUT, ":utf8";
Dump $mixedstring;
print "UTF IS ON\n" if utf8::is_utf8($mixedstring);
print "LENGTH DIFFERS\n" if length($mixedstring) != bytes::length($mix
+edstring);
open my $f, ">", "test2";
binmode $f;
syswrite $f, $mixedstring or die;
print "ALL OK\n";
__END__
SV = PV(0x1d6eb48) at 0x1c7fab8
REFCNT = 1
FLAGS = (PADMY,POK,pPOK,UTF8)
PV = 0x1d79820 "123\303\221\302\204\303\221\302\204\303\221\302\204\
+303\221\302\204"\0 [UTF8 "123\x{d1}\x{84}\x{d1}\x{84}\x{d1}\x{84}\x{d
+1}\x{84}"]
CUR = 19
LEN = 24
UTF IS ON
LENGTH DIFFERS
ALL OK
After __END__ of each file I appended program output.
Let's ignore for the moment the fact that those strings completely different and contains different characters and the fact that at some point of time one of the strings was interpreted as latin-1 etc
So, in both cases strings have UTF-8 bit set. They both have non ASCII-7bit octets. Their length() and bytes::length differs. And I expect those strings should behave same way
Question is why in one case string was treated as 'wide character string' and syswrite terminated the program. In other case all was working fine
p.s reproduced on perl 5.10 and perl 5.14 (linux)
UPD: escaped utf chars in sourcecode, as perlmonks eats it
UPD: SOLVED: http://www.perlmonks.org/?node_id=1032996 http://www.perlmonks.org/?node_id=1033006
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.