in reply to length() miscounting UTF8 characters?
Here's a Perl script that counts the number of bytes, code points, and graphemes in each UTF-8 encoded word. It also tallies the code points by Unicode blocks.
#!perl use v5.14; use strict; use warnings; use utf8; use Encode qw( encode_utf8 ); use Unicode::UCD qw( charblock ); binmode STDOUT, ':encoding(UTF-8)'; while (my $word = <DATA>) { chomp $word; my $length_in_bytes = length_in_bytes($word); my $length_in_code_points = length_in_code_points($word); my $length_in_graphemes = length_in_graphemes($word); my $code_points_in_blocks = code_points_in_blocks($word); printf "%-12s | Bytes: %2d | Code Points: %2d | Graphemes: %2d | Blocks: %s\n", $word, $length_in_bytes, $length_in_code_points, $length_in_graphemes, $code_points_in_blocks; } exit 0; sub length_in_bytes { my $word = shift; my $length = length encode_utf8($word); return $length; } sub length_in_code_points { my $word = shift; my $length = length $word; return $length; } sub length_in_graphemes { my $word = shift; my $length = () = $word =~ m/\X/g; return $length; } sub code_points_in_blocks { my $word = shift; my %total_code_points_by; my $blocks = ''; for my $character (split m//, $word) { my $block = charblock(ord $character); $total_code_points_by{$block}++; } for my $block (sort keys %total_code_points_by) { my $total = $total_code_points_by{$block}; $blocks .= sprintf "%s%s (%d)", (length $blocks ? ', ' : ''), $block, $total; } return $blocks; } __DATA__ æ æð æða æðaber æðahnútur æðakölkun æðardúnn æðarfugl æðarkolla æðarkóngur æðarvarp æði æðimargur æðisgenginn æðiskast æðislegur æðrast æðri æðrulaus æðruleysi æðruorð æðrutónn æðstur æður æfa
Here's the output of the script.
æ | Bytes: 2 | Code Points: 1 | Graphemes: 1 | Blocks: Latin-1 Supplement (1) æð | Bytes: 4 | Code Points: 2 | Graphemes: 2 | Blocks: Latin-1 Supplement (2) æða | Bytes: 5 | Code Points: 3 | Graphemes: 3 | Blocks: Basic Latin (1), Latin-1 Supplement (2) æðaber | Bytes: 8 | Code Points: 6 | Graphemes: 6 | Blocks: Basic Latin (4), Latin-1 Supplement (2) æðahnútur | Bytes: 12 | Code Points: 9 | Graphemes: 9 | Blocks: Basic Latin (6), Latin-1 Supplement (3) æðakölkun | Bytes: 12 | Code Points: 9 | Graphemes: 9 | Blocks: Basic Latin (6), Latin-1 Supplement (3) æðardúnn | Bytes: 11 | Code Points: 8 | Graphemes: 8 | Blocks: Basic Latin (5), Latin-1 Supplement (3) æðarfugl | Bytes: 10 | Code Points: 8 | Graphemes: 8 | Blocks: Basic Latin (6), Latin-1 Supplement (2) æðarkolla | Bytes: 11 | Code Points: 9 | Graphemes: 9 | Blocks: Basic Latin (7), Latin-1 Supplement (2) æðarkóngur | Bytes: 13 | Code Points: 10 | Graphemes: 10 | Blocks: Basic Latin (7), Latin-1 Supplement (3) æðarvarp | Bytes: 10 | Code Points: 8 | Graphemes: 8 | Blocks: Basic Latin (6), Latin-1 Supplement (2) æði | Bytes: 5 | Code Points: 3 | Graphemes: 3 | Blocks: Basic Latin (1), Latin-1 Supplement (2) æðimargur | Bytes: 11 | Code Points: 9 | Graphemes: 9 | Blocks: Basic Latin (7), Latin-1 Supplement (2) æðisgenginn | Bytes: 13 | Code Points: 11 | Graphemes: 11 | Blocks: Basic Latin (9), Latin-1 Supplement (2) æðiskast | Bytes: 10 | Code Points: 8 | Graphemes: 8 | Blocks: Basic Latin (6), Latin-1 Supplement (2) æðislegur | Bytes: 11 | Code Points: 9 | Graphemes: 9 | Blocks: Basic Latin (7), Latin-1 Supplement (2) æðrast | Bytes: 8 | Code Points: 6 | Graphemes: 6 | Blocks: Basic Latin (4), Latin-1 Supplement (2) æðri | Bytes: 6 | Code Points: 4 | Graphemes: 4 | Blocks: Basic Latin (2), Latin-1 Supplement (2) æðrulaus | Bytes: 10 | Code Points: 8 | Graphemes: 8 | Blocks: Basic Latin (6), Latin-1 Supplement (2) æðruleysi | Bytes: 11 | Code Points: 9 | Graphemes: 9 | Blocks: Basic Latin (7), Latin-1 Supplement (2) æðruorð | Bytes: 10 | Code Points: 7 | Graphemes: 7 | Blocks: Basic Latin (4), Latin-1 Supplement (3) æðrutónn | Bytes: 11 | Code Points: 8 | Graphemes: 8 | Blocks: Basic Latin (5), Latin-1 Supplement (3) æðstur | Bytes: 8 | Code Points: 6 | Graphemes: 6 | Blocks: Basic Latin (4), Latin-1 Supplement (2) æður | Bytes: 6 | Code Points: 4 | Graphemes: 4 | Blocks: Basic Latin (2), Latin-1 Supplement (2) æfa | Bytes: 4 | Code Points: 3 | Graphemes: 3 | Blocks: Basic Latin (2), Latin-1 Supplement (1)
UPDATE: If you add these three words to the end of the list in the __DATA__ block of the the UTF-8 encoded Perl script…
한국말 piñón piñón
…then the report will include these three lines…
한국말 | Bytes: 9 | Code Points: 3 | Graphemes: 3 | Blocks: Hangul Syllables (3) piñón | Bytes: 7 | Code Points: 5 | Graphemes: 5 | Blocks: Basic Latin (3), Latin-1 Supplement (2) piñón | Bytes: 9 | Code Points: 7 | Graphemes: 5 | Blocks: Basic Latin (5), Combining Diacritical Marks (2)
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^2: length() miscounting UTF8 characters?
by AppleFritter (Vicar) on Apr 28, 2014 at 09:37 UTC | |
by Jim (Curate) on Apr 28, 2014 at 18:39 UTC |
In Section
Seekers of Perl Wisdom