length() normally deals in logical characters, not physical bytes

{
    package LenTestC;
    use Encode;
    sub new {
        my $class = shift;
        my $self = '';
        return bless \$self, $class;
    }
    sub add {
        my ($self, $data) = @_;
        $$self .=  Encode::encode_utf8($data);
    }
    sub len {
        my $self = shift;
        return length $$self;
    }
}
[download]

Note that your print/tell solution did the same kind of accounting, reporting bytes instead of characters.

Many thanks for your valuable answer, I reproduced the same performance gains on my system, while not grasping the why.

I was told that since utf-8 string management was natively integrated into Perl core a string has an internal flag to tell if it is utf-8 or not.

When parsing an xml file declared as Encoding="utf-8", the strings parsed by the XML SAX Parser are not given in utf-8? (I did not noticed that because I do not display processed data, so if the string is given undecoded i guess it is written as-is, but I should double-check that)
I seem to understand that the SAX writer might query many times the data size, so the overweight of encoding the data is compensated by the many calls to a length() that has better performance.
But I do not see how length() is different depending on the string encoding: if the string is not in utf-8 length should return a byte size (1 byte per character) while on utf-8 we must process each byte to know if it is a simple char, a starting byte of a multi-byte char, or a continuation byte of a multi-byte char. I would have think that processing an utf-8 string has worse performance than a plain string...


Note that your print/tell solution did the same kind of accounting, reporting bytes instead of characters.

Yes, but that is not a problem as I am asked to split the xml on a file size basis (per 30, 100 or 200 MB chunks) so counting the bytes is ok.

Encoding can be a challenge to get one's head around. When you read the strings in from your XML parsing, Perl pulls them in as a series of UTF-8 characters, and the string that contains them has the UTF-8 flag set to true. In order to determine the length of the string, each byte must be queried to determine to figure out how many characters are represented, thus the slow length.

Invoking Encode::encode_utf8($data) returns the UTF-8 string transformed into the equivalent byte stream. Essentially, from Perl's perspective, it breaks the logical connection between the bytes, and leaves it as some combination of high bit and low bit characters. Now, since every record in the string is exactly 1 byte wide, the byte count requires no introspection.

So:

print length chr 199;
[download]

outputs 1 while

use Encode; 
print length Encode::encode_utf8(chr 199);
[download]

outputs 2. Similarly, if you run

say join ",", map ord, split //, chr 199;
[download]

you output 199, while

use Encode; 
say join ",", map ord, split //, Encode::encode_utf8(chr 199);
[download]

outputs 195, 135.

However, if your terminal is set to display UTF-8, printing both of those strings will output the same because the series of bits is unaffected.

Does that help?

---

#11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.

5-20 times slower is mentioned here Speed

It seems to me that bigger is the string, longer is the time to get its size, (exponentialy?).

while (start_ptr < end_ptr) {
    start_ptr += UTF8SKIP(start_ptr);
    len++;
}
[download]

Your LenTestA is exponential because it basically does

PL_utf8_length($chunk)                    # first iteration
PL_utf8_length($chunk . $chunk)           # second iteration
PL_utf8_length($chunk . $chunk . $chunk)  # third iteration
...and so on...
[download]

PL_utf8_length($chunk)  # 1st
PL_utf8_length($chunk)  # 2nd
PL_utf8_length($chunk)  # 3rd
...
[download]

Your LenTestA is exponential

That would be quadratic growth. You have a series of operations that grows linearly, and your number of invocations grows linearly with line length.

---

#11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.

Maybe, maybe, I don't speak english very well. Basically, it's O(N**2).