Re: Fixing broken character encoding


Come for the quick hacks, stay for the epiphanies.
	PerlMonks

Re: Fixing broken character encoding

by Anonymous Monk

on Jul 26, 2012 at 03:02 UTC ( #983760=note: print w/replies, xml )

Need Help??

in reply to Fixing broken character encoding

It is possible if its been malformed once (single step), multiple iterations can be impossible.

IIRC I think http://validator.w3.org/ can help ( Bundle::W3C::Validator )

As can these
HTML::Encoding - Determine the encoding of HTML/XML/XHTML documents
Encode::Detective - detect a data encoding
Encoding::FixLatin - takes mixed encoding input and produces UTF-8 output
Encode::DoubleEncodedUTF8 - Fix double encoded UTF-8 bytes to the correct one

But you ought to post some minimal html

I figure it ought to be as simple as parsing the file, decoding the entities, treating the string as octets and deciding what charset it is

Comment on Re: Fixing broken character encoding

Replies are listed 'Best First'.

Re^2: Fixing broken character encoding
by Anonymous Monk on Jul 26, 2012 at 04:04 UTC

This could work

#!/usr/bin/perl --
use strict; use warnings;
use Data::Dump        qw' dd                              ';
use HTML::Entities    qw' encode_entities decode_entities ';
use Encode            qw' encode decode                   ';
use Encode::Detective qw' detect                          ';

my $odata = my $str = '&acirc;&#128;&#153;';

decode_entities($str);
dd $str;
dd encode_entities($str);

my $encoding = detect($str);
dd $encoding;

$str = decode( 'UTF-8', $str );
dd $str;
dd encode_entities($str);

__END__
"\xE2\x80\x99"
"&acirc;&#128;&#153;"
"UTF-8"
"\x{2019}"
"&rsquo;"
[download]

[reply]
[d/l]

Re^3: Fixing broken character encoding

by pfaut (Priest) on Jul 26, 2012 at 10:25 UTC

That's showing some promise. Thank you.

90% of every Perl application is already written. ⇒

dragonchild

[reply]

In Section Seekers of Perl Wisdom