Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re^2: Chinese site and decoded_content() trouble

by isync (Hermit)
on Jun 11, 2007 at 08:50 UTC ( [id://620442]=note: print w/replies, xml ) Need Help??


in reply to Re: Chinese site and decoded_content() trouble
in thread Chinese site and decoded_content() trouble

Thanks for your comment! And thank you for pointing out the raise_error switch - which I wasn't aware of (rtfm...)!

Running your test script gives me these errors:
test: http://cn.life.dada.net/people/ is decoded content utf8? 1 test: http://www.sina.com.cn/ decode failed: euc-cn "\x82" does not map to Unicode at /usr/lib/perl/ +5.8/Encode.pm line 166. at test_ku6.pl line 22 test: http://www.ku6.com/show/34D6sgY4X6w3YegR.html decode failed: utf8 "\xE6" does not map to Unicode at /usr/lib/perl/5. +8/Encode.pm line 166. at test_ku6.pl line 22 test: http://www.xinhua.cn/ decode failed: euc-cn "\xA9" does not map to Unicode at /usr/lib/perl/ +5.8/Encode.pm line 166. at test_ku6.pl line 22
which points me at the problem that decoded_content uses the wrong charset for decoding. (right?)

A search on the issue led me to this helpful thread http://www.issociate.de/board/post/400895/Fetching_the_charset_when_set_in_meta..html and the non-cpan module HTTP::Response::Charset (which I haven't tried so far).

So I am stuck here with three options:
1. use Encode::Guess
2. use HTML::Encoding
3. use HTTP::Response::Charset

A quick test seems to show that HTML::Encoding is the most reliable (and much less a hassle to install than Encode::Detect). Now I will try
my $enco = encoding_from_http_message($resp); my $utf8 = decode($enco => $resp->content);
But how do I combine it elegantly with LWP? (I am not a great module guru..)
Is there a way to pull it into the LWP context and get, let's say, $response->as_utf8 (which is the result of the above)?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://620442]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others musing on the Monastery: (4)
As of 2024-04-24 06:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found