the perldoc entry for length (which I checked
beforehand to make sure it wouldn't count bytes -- hence my
confusion)
It "normally deals in logical characters", but its logic doesn't
cover all the intricacies of unicode.
Do you have any specific languages or complex data in mind with which it might fail?
Yes, Thai language is the main one I'm involved with. The modified
script below shows that length counts diacriticals in
Thai, which may or may not be what is wanted, and is inconsistent
with the results for Latin diacriticals in your dataset, which
length isn't counting separately. I'm using
pre tags so that the Thai will display correctly and shortened lines to facilitate copy/paste.
#!/usr/bin/env perl
use warnings;
use v5.14;
use Unicode::Normalize qw/NFD/;
binmode STDOUT, 'utf8';
binmode DATA, 'encoding(utf-8)';
while (<DATA>) {
chomp;
print $_, ': ';
s/[A-Za-z]//g;
my $alphacount = () = /\p{Alpha}/g;
say "non-(A-Za-z) symbols <$_>",
" contain $alphacount",
" alphabetic characters and ",
getdia($_), " diacritical chars.";
say "length() thinks there are ",
length, " characters\n";
}
sub getdia {
my $normalized = NFD($_[0]);
my $diacount = () =
$normalized =~ /\p{Dia}/g;
return $diacount;
}
__DATA__
เป็น
ผู้หญิง
เมื่อวันก่อน
æðaber
æðahnútur
æðakölkun
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|