5.8.0 and 5.8.8 both return the same result for me.

is tagged as a unicode string
[download]

This is perl, v5.8.0 built for MSWin32-x86-multi-thread
Binary build 806 provided by ActiveState Corp.
Built 00:45:44 Mar 31 2003

Encode 1.83
[download]

This is perl, v5.8.8 built for MSWin32-x86-multi-thread
Binary build 817 [257965] provided by ActiveState
Built Mar 20 2006 17:54:25

Encode 2.12
[download]

It would make no sense for it not to be tagged. When one asks to decode a string of bytes (UTF8 off) to a string of chars (UTF8 on), it makes no sense that the same call sometimes returns a string of chars and sometimes get a string of bytes.

I'd say the bug is in the docs.

Except there is the issue of efficiency (see my other post above). Representing a string of characters which all happen to lie within the ASCII range as an untagged byte string allows the byte-oriented regex engine to be used.

It's a very similar idea to using machine words to hold integers up to a certain value, and then switching to a different representation for bignums. It doesn't make a difference to correctness, but it does make a difference to performance.

To repeat what i said elsewhere, IMO this is a bug that should be reported.

Id do it for you, but using perlbug on win32 is a real PITA.

---
$world=~s/war/peace/g

It shouldn't really matter anyway. ASCII => utf8.

Update: it does the same for me. perl 5.8.8, Encode 2.12.

There's also this in the docs:

CAVEAT: When you run $string = decode("utf8", $octets), then $string m
+ay not be equal to $octets. Though they both contain the same data, t
+he utf8 flag for $string is on unless $octets entirely consists of AS
+CII data (or EBCDIC on EBCDIC machines). See "The UTF-8 flag" below.
[download]

and the difference does matter from a performance point of view.

UTF-8 tagged values in perl are contagious - concatenation with an untagged value will result in a tagged value (all well and good). But the regex engine on unicode strings is slower than on byte strings.

Basically with this change in behaviour you can lose performance in a utf8-aware-and-correct application which has the vast majority of its inputs in ASCII, since the previously uncommon case of handling unicode strings is now the 100% case.

This isn't theoretical, I'm fighting a significant CPU cost increase, which adds up over many servers.

I understand your concern, but I'm still trying understand why the OP question comes up in the context of your app.

Are you getting actual utf8 data from a file handle that does not use the ":utf8" PerlIO layer (so that perl begins by assuming it's just a raw byte stream)? And if that's the case, are you trying to work out a way to use "byte-semantics" regexen where possible, and "character-semantics" only when necessary?

If that's your situation, here's an easy, low-cpu-load method to check whether a raw byte string needs be tagged as utf8:

   if ( length($string) > $string =~ tr/\x00-\x7f// ) {
      $string = decode( 'utf8', $string );
   }
[download]

(updated as per fenLisesi's reply -- thanks!)

Or, given that the original string is not tagged as a perl-internal utf8 scalar value (utf8 flag is off), this might be just as good or better:

   if ( $string =~ /[\x80-\xff]/ ) {
      $string = decode( 'utf8', $string );
   }
[download]

I'm not actually sure whether one way is faster than the other, or whether the relative speed would depend on your data; "length()" and "tr///" are both pretty fast whereas a regex match is slower, but tr always processes the whole string, whereas that regex match will stop at the first non-ascii byte.

use Encode;
my $t = "abc";
my $d = decode('ascii',$t);
printf "is_utf8=%d\n", (utf8::is_utf8($d) ? 1 : 0);
[download]

is_utf8=1
[download]

Bryce Nesbitt, Berkeley Electronic Press, Berkeley CA