Unexpected utf8 in hash keys

kappa has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Unexpected utf8 in hash keys by kappa (Chaplain) on Feb 20, 2008 at 11:27 UTC
Fat comma is really at fault: `use utf8; sub ff { die if utf8::is_utf8($_[0]); } ff(asd => 1);` [download] --kap	[reply] [d/l]
Re^2: Unexpected utf8 in hash keys (encoding) by ikegami (Patriarch) on Feb 20, 2008 at 12:57 UTC
encoding produces the opposite result. `#use encoding 'UTF-8'; # or 'utf8' #use utf8; use Encode qw( is_utf8 ); sub ff { print is_utf8($_[0]) ? 1 : 0, "\n"; } # none enco utf8 both # ---- ---- ---- ---- ff("asd"); # 0 1 0 1 ff('asd'); # 0 1 0 1 ff(qw(asd)); # 0 1 0 1 ff(asd => 1); # 0 0 1 1 { no strict; ff(asd); } # 0 0 1 1 ff(-asd); # 0 0 1 1 ff(asd::); # 0 0 1 1` [download]	[reply] [d/l]
Re^2: Unexpected utf8 in hash keys (force_word) by ikegami (Patriarch) on Feb 20, 2008 at 12:45 UTC
After some investigation, barewords in general (anything parsed by `force_word`) are affected. `use utf8; sub ff { print utf8::is_utf8($_[0]) ? 1 : 0, "\n"; } ff("asd"); # 0 ff('asd'); # 0 ff(qw(asd)); # 0 ff(asd => 1); # 1 { no strict; ff(asd); } # 1 ff(-asd); # 1 ff(asd::); # 1` [download] I think function names (including built-ins) and keywords are also affected. Gotta go. No time to devise a fix atm.	[reply] [d/l] [select]
Re^2: Unexpected utf8 in hash keys (bug) by ikegami (Patriarch) on Feb 20, 2008 at 12:03 UTC
I'd expect both of the following calls to output the same thing, so I think it's a bug. `use utf8; sub ff { print utf8::is_utf8($_[0]) ? 1 : 0, "\n"; } ff(asd => 1); # 1 ff("asd"); # 0` [download] Tested in 5.8.8 and 5.10.0.	[reply] [d/l]
Re: Unexpected utf8 in hash keys by graff (Chancellor) on Feb 20, 2008 at 13:57 UTC
It's definitely worthwhile to know and understand the trickiness demonstrated so clearly by ikegami's various tests, and I would agree that some of his results point to "actionable" inconsistencies that should probably be treated as bugs. BUT... when you say: It cost us a lot of blood and sweat to debug why some perfectly ASCII strings would suddenly get the flag. Does this mean you were using the utf8 flag to determine whether or not a string contains wide characters? That is not what the flag is for, and you shouldn't be using it that way. To test for wide characters in a string, use a regex: `if ( /[^[:ascii:]]/ ) { ... } # which is equivalent to if ( /[^\x00-\x7f]/ ) { ... }` [download] The purpose of the utf8 flag, as I understand it, is to answer the question: if there happen to be non-ASCII bytes in this string, are they to be interpreted as utf8 characters, or not? The treatment of an all-ASCII string should be the same regardless of whether the utf8 flag is set.	[reply] [d/l]
Re^2: Unexpected utf8 in hash keys by kappa (Chaplain) on Feb 20, 2008 at 15:06 UTC
Yes, it was a wrong way to do that -- an attempt based on wrong guess that perl would not set utf8 flag on ASCII strings. --kap	[reply]
Re: Unexpected utf8 in hash keys by pc88mxer (Vicar) on Feb 20, 2008 at 14:13 UTC
Actually, the issue really is the `use utf8` which allows you to use utf8 in your program identifiers. For instance: `use strict; use warnings; my %hash = ( asd => 1 ); sub ff { print utf8::is_utf8($_[0]) ? 1 : 0, "\n"; } eval { use utf8; ff(%hash); # now prints 0 };` [download] I suppose it's reasonable that perl encodes barewords as utf8 if you `use utf8` even if only ascii characters are involved. Update: from the `utf8` documentation: `The "use utf8" pragma tells the Perl parser to allow UTF-8 in the pro- gram text in the current lexical scope (allow UTF-EBCDIC on EBCDIC based platforms). ... Do not use this pragma for anything else than telling Perl that your script is written in UTF-8. The utility functions described below are useful for their own purposes, but they are not really part of the "pragmatic" effect.` [download]	[reply] [d/l] [select]
Re^2: Unexpected utf8 in hash keys by kappa (Chaplain) on Feb 20, 2008 at 15:01 UTC
We actually use lots of non-ASCII strings so we need `use utf8`. --kap	[reply] [d/l]
Re: Unexpected utf8 in hash keys by doc_faustroll (Scribe) on Feb 20, 2008 at 18:19 UTC
I may be reading this too quickly and not appreciating or understanding your intent, or what exactly you are using the use utf8 pragma for or why. My hunch is that you might benefit from not enquiring as to when the utf8 flag is set or not. Can you tell me your purposes? We can play around with the internals of Perl and how it represents strings all day long, but I gather you have more than academic interest here? This may be a place where solid pragmatism trumps. Have you read perlunitut?	[reply]
Re: Unexpected utf8 in hash keys by Juerd (Abbot) on Feb 21, 2008 at 01:42 UTC
Out of curiosity, why does the flag bother you?	[reply]
Re^2: Unexpected utf8 in hash keys by kappa (Chaplain) on Feb 26, 2008 at 18:29 UTC
We are in the process of converting a huge production codebase from 5.6 + Unicode::String to 5.8. This is as painful as it can get and we log this flag in a lot of places just to find codepaths that need attention. --kap	[reply]
Re: Unexpected utf8 in hash keys by creamygoodness (Curate) on Aug 27, 2009 at 11:12 UTC
Does anyone know what the performance penalty of utf8 hash keys -- even if they contain only ASCII chars -- is? Benchmarking script: Read more... (2 kB) Results for vanilla custom-compiled Perl 5.10.0 on Mac OS X: Read more... (2 kB)	[reply] [d/l] [select]


Keep It Simple, Stupid
	PerlMonks