Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re^3: How to handle encodings?

by clinton (Priest)
on Mar 07, 2009 at 19:06 UTC ( #749078=note: print w/replies, xml ) Need Help??

in reply to Re^2: How to handle encodings?
in thread How to handle encodings?

A typical place where Encode::Guess falls down (through no fault of its own) is in differentiating one variant of iso-8859 from another.

Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

Without prior knowledge, you're up the creek without a paddle. So I agree wholeheartedly with Moritz's suggestion of converting everything to UTF8, while you still know what encoding it is in.

(graff - I know you're too wise a monk to have been suggesting otherwise, but I wanted to provide a simple example of just how limited Encode::Guess can be.)


Replies are listed 'Best First'.
Re^4: How to handle encodings?
by Narveson (Chaplain) on Mar 07, 2009 at 22:28 UTC
    Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

    Any moderately experienced human eye, given a paragraph of context. Have you ever played the game of pulling down the Character Encoding menu in your Web Browser and trying the same page in different encodings?

    This is not a dissent from moritz's and clinton's advice. By all means heed any available declarations.

    But in the absence of a declaration, the right encoding is humanly guessable, even when the alternatives are different members of the ISO-8859 series. And if it's humanly guessable, then a CPAN module should in principle be able to do it too.

    I don't know what algorithm Encode::Guess uses. My own algorithm (untested) would be

    • submit a phrase to Google as a search term;
    • accept Google's proposed spelling correction.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://749078]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (2)
As of 2022-10-07 05:47 GMT
Find Nodes?
    Voting Booth?
    My preferred way to holiday/vacation is:

    Results (29 votes). Check out past polls.