in reply to Re: Why does Encode::Repair only correctly fix one of these two tandem characters?
in thread Why does Encode::Repair only correctly fix one of these two tandem characters?
You're spot on, Bethany. Thanks.
The byte \x9D is being converted to the Unicode character U+FFFD REPLACEMENT CHARACTER (EF BF BD) upstream. So the question now is: What's special about \x9D that isn't special about \x9C?* Hmm…
I added statements to the demonstration script to display a hex dump of the UTF-8 double-encoded bytes:
use charnames qw( :full ); use Encode qw( encode decode ); use Encode::Repair qw( repair_double ); binmode STDOUT, ':encoding(UTF-8)'; my $ldqm = "\N{LEFT DOUBLE QUOTATION MARK}"; my $rdqm = "\N{RIGHT DOUBLE QUOTATION MARK}"; $ldqm = encode('UTF-8', decode('Windows-1252', encode('UTF-8', $ldqm)) +); $rdqm = encode('UTF-8', decode('Windows-1252', encode('UTF-8', $rdqm)) +); say join ' ', map { sprintf '%02X', $_ } unpack 'C*', $ldqm; say join ' ', map { sprintf '%02X', $_ } unpack 'C*', $rdqm; say repair_double($ldqm, { via => 'Windows-1252' }); say repair_double($rdqm, { via => 'Windows-1252' }); __END__
C3 A2 E2 82 AC C5 93 C3 A2 E2 82 AC EF BF BD “ ��?
*UPDATE: The short answer to the question is that 9C is a defined character in the Windows-1252 character encoding ('œ') and 9D is not.
In Section
Seekers of Perl Wisdom