Re: The “real length" of UTF8 strings

in reply to The “real length" of UTF8 strings

For any UTF8 string, we have four "lengths":

the length in codepoints:

perl -C63 -MDevel::Peek -Mutf8 -le '$_="(\x{5fcd} Guimarăes)"; Dump($_
+); print length($_); print'
SV = PV(0x8154b00) at 0x8153bd4
  REFCNT = 1
  FLAGS = (POK,pPOK,UTF8)
  PV = 0x8170460 "(\345\277\215 Guimar\303\243es)"\0 [UTF8 "(\x{5fcd} 
+Guimar\x{e3}es)"]
  CUR = 16
  LEN = 20
13
[download]

(忍 Guimarăes)

the length in graphemes (the "a" is one, the composing "~" is another):

perl -C63 -MDevel::Peek -Mutf8 -MUnicode::Normalize -le '$_="(\x{5fcd}
+ Guimarăes)"; $_ = NFD $_; Dump($_); print length($_); print'
SV = PV(0x8154b00) at 0x8153bd4
  REFCNT = 1
  FLAGS = (POK,pPOK,UTF8)
  PV = 0x816feb8 "(\345\277\215 Guimara\314\203es)"\0 [UTF8 "(\x{5fcd}
+ Guimara\x{303}es)"]
  CUR = 17
  LEN = 20
14
[download]

(忍 Guimarães)

the length in columns of text used (the string has one wide character):

perl -C63 -MDevel::Peek -Mutf8 -mText::CharWidth=mbswidth -le '$_="(\x
+{5fcd} Guimarăes)"; Dump($_); print mbswidth($_); print'
SV = PV(0x8154b00) at 0x8153bd4
  REFCNT = 1
  FLAGS = (POK,pPOK,UTF8)
  PV = 0x8170460 "(\345\277\215 Guimar\303\243es)"\0 [UTF8 "(\x{5fcd} 
+Guimar\x{e3}es)"]
  CUR = 16
  LEN = 20
14
[download]

(忍 Guimarăes)

the length in bytes of the string (notice I didn't print the string after the encode):

perl -C63 -MDevel::Peek -Mutf8 -mEncode=encode_utf8 -le '$_="(\x{5fcd}
+ Guimarăes)"; Dump($_); print length(encode_utf8 $_)'
SV = PV(0x8154b00) at 0x8153bd4
  REFCNT = 1
  FLAGS = (POK,pPOK,UTF8)
  PV = 0x8170460 "(\345\277\215 Guimar\303\243es)"\0 [UTF8 "(\x{5fcd} 
+Guimar\x{e3}es)"]
  CUR = 16
  LEN = 20
16
[download]

[]s, HTH, Massa (κς,πμ,πλ)

In Section Seekers of Perl Wisdom