http://qs321.pair.com?node_id=812656


in reply to Re: Wierd behaviour with HTML::Entities::decode_entities()
in thread Wierd behaviour with HTML::Entities::decode_entities()

One can imagine a strange scenario where,

If the scenario you describe is possible, it's a bug in HTML::Entities. And it's not what the OP is seeing. The OP claims he need to do too many decodings, not too few.

I think you're thinking of double-encoding, where "foo" was accidentally encoded as

"foo"
when it should have been encoded as
"foo"

Replies are listed 'Best First'.
Re^3: Wierd behaviour with HTML::Entities::decode_entities()
by JadeNB (Chaplain) on Dec 14, 2009 at 11:29 UTC
    If the scenario you describe is possible, it's a bug in HTML::Entities. And it's not what the OP is seeing. The OP claims he need to do too many decodings, not too few.

    I don't think that I'm claiming what you think I'm claiming. :-) What I meant was that, say, 
 (or even just 
) would be interpreted (incorrectly) as 
 by two passes of decode_entities, but not by one. This gives “unexpected decoding” after the second pass, but it's not a bug in HTML::Entities.

    (UPDATE: I meant what I meant, but it wasn't quite what I said. A better example is a, which becomes a after one pass of the decoder and then (incorrectly) a after another. This is very particular to the ordering I mentioned earlier (first decimal, then hexadecimal, then named entities are expanded). This is the ordering in the pure-Perl decode_entities_old in HTML::Entities; I have no idea if the XS version also behaves this way. Perhaps you thought that I was mentioning that, say, &#amp;quot; " would be incorrectly converted to "? You're right, it seems to me that that is what will happen, and that it is a bug.)

    On the other hand, I couldn't, and can't, think of a way that this would give the behaviour that the OP is seeing. The kind of double-encoding you mentioned sounds far more likely—and the remedy, I think, is the same, to look at the intermediate steps along the way to see where something's going wrong. (Actually, I guess that's so generic that it's true for just about any problem.)

      This is very particular to the ordering I mentioned earlier (first decimal, then hexadecimal, then named entities are expanded)

      No, it isn't

      Different nesting order:

      >perl -MHTML::Entities -le"print decode_entities '"'; " >perl -MHTML::Entities -le"print decode_entities '"'; "

      Different sibling order:

      [ Can't find a valid example ]

      Or I still don't understand. Please given an example where ordering matters.

        The source for decode_entities_old, which I thought was just a pure-Perl version of decode_entities, looks roughly like this (I've stripped out some context dependence):

        sub decode_entities_old { my $array = [ @_ ]; my $c; for (@$array) { s/(&\#(\d+);?)/$2 < 256 ? chr($2) : $1/eg; s/(&\#[xX]([0-9a-fA-F]+);?)/$c = hex($2); $c < 256 ? chr($c) : $1/ +eg; s/(&(\w+);?)/$entity2char{$2} || $1/eg; } return @$array; }
        With this code, and the under-populated hash my %entity2char = ( amp => '&', quot => '"' ), we have
        say decode_entities_old '&#x26;amp;quot;'; => &quot; say decode_entities_old '&amp;#x26;quot;'; => &#x26;quot;
        We would get (essentially) the opposite behaviour if we switched the second and third substitutions; that's what I mean by “This is very particular to the ordering”.

        Of course, your example shows that decode_entities (which I guess is implemented in XS—I couldn't find it in the source) doesn't exhibit this buggy behaviour, so I guess that there's a reason that the sub I quoted has _old postpended. :-)

        UPDATE: I just noticed that I'd mangled my intended bug-inducer, &&#x71;uot;, above. I wonder if decode_entities (not decode_entities_old) handles it correctly?