http://qs321.pair.com?node_id=1096817


in reply to Re: Why does Encode::Repair only correctly fix one of these two tandem characters?
in thread Why does Encode::Repair only correctly fix one of these two tandem characters?

You're spot on, Bethany. Thanks.

The byte \x9D is being converted to the Unicode character U+FFFD REPLACEMENT CHARACTER (EF BF BD) upstream. So the question now is:  What's special about \x9D that isn't special about \x9C?* Hmm…

I added statements to the demonstration script to display a hex dump of the UTF-8 double-encoded bytes:

use charnames qw( :full ); use Encode qw( encode decode ); use Encode::Repair qw( repair_double ); binmode STDOUT, ':encoding(UTF-8)'; my $ldqm = "\N{LEFT DOUBLE QUOTATION MARK}"; my $rdqm = "\N{RIGHT DOUBLE QUOTATION MARK}"; $ldqm = encode('UTF-8', decode('Windows-1252', encode('UTF-8', $ldqm)) +); $rdqm = encode('UTF-8', decode('Windows-1252', encode('UTF-8', $rdqm)) +); say join ' ', map { sprintf '%02X', $_ } unpack 'C*', $ldqm; say join ' ', map { sprintf '%02X', $_ } unpack 'C*', $rdqm; say repair_double($ldqm, { via => 'Windows-1252' }); say repair_double($rdqm, { via => 'Windows-1252' }); __END__
C3 A2 E2 82 AC C5 93
C3 A2 E2 82 AC EF BF BD
“
��?

*UPDATE:  The short answer to the question is that 9C is a defined character in the Windows-1252 character encoding ('œ') and 9D is not.