Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much

Re^3: UTF-8 and Unicode the hard way

by haj (Priest)
on May 09, 2022 at 20:51 UTC ( #11143718=note: print w/replies, xml ) Need Help??

in reply to Re^2: UTF-8 and Unicode the hard way
in thread UTF-8 and Unicode the hard way

Corion provided the correct answer, but you failed to verify it. If you print a decoded non-ASCII character, then you get the wide character warning. This is exactly what happens when you print the result of your own substitutions:

$ perl -E "print qq(\x{100})"
Wide character in print at -e line 1.

Printed output needs to be encoded into a byte stream which the receiving side is able to understand. In many cases like contemporary Unix terminals, UTF-8 is a good guess - which is the encoding your $answer came from.

Replies are listed 'Best First'.
Re^4: UTF-8 and Unicode the hard way
by Anonymous Monk on May 10, 2022 at 17:26 UTC

    (OP here -- sorry, I should have obtained a username before starting this)

    Sorry also for the delay, I stopped to do more tests based on the useful information you've all given me. The "wide character" error happens if I'm trying to decode a string that already has what I have been calling "unicode" extended characters in it. (Remember I said this is all a great mystery to me and I'd really like it to all go away forever? That includes my incorrect terminology.) That is, if it's already got characters such as \x{103}, trying to decode them will produce that error. This turns out to be because one of my data sources sends extended characters in one format and one in a different format (this is an API that has to merge data from several sources for a single output stream).

    Or, more concretely: One of my data sources sends lower-case-a-with-breve ă as \xc4\x83, which is the kind that does need translating for my purposes, and the other data source sends it as \x{103}, which for my purposes is already translated into the format I need. decode('UTF-8') works properly on the former and errors on the latter, which seems to be correct behavior based on what you've said. I didn't realize the two data sources were doing it differently (neither of them has any documentation of what they do, alas) and I picked the wrong horse for my previous test.

    The reason I was calling the former of those "UTF-8" and the latter "Unicode" was because of pages like, where I look up characters for when I have to translate them by brute force ... I'm still not sure of the correct term for the longer Unicode encoding where ă is \x{103} (AKA U+0103).

    Anyway, thank you! It does look like decode() does the right thing, when its user isn't dumb.

      For your given example, this utility might prove illuminating. You can see that the character you describe has the hex code point 0103 and is constructed of the hex bytes c483. This is what each of the constructions in your 2 data sources are referring to. You will have to treat the two sources differently if you want to handle them both successfully.


        Fortunately, I don't have to handle the second data source at all; it's already in the format I need. But thank you, that's a useful tool.
      The writeup tips do say <i> tags are allowed, so I have no idea why it tried to make that first italicized paragraph into a node link. Sorry about that.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11143718]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (2)
As of 2023-03-31 02:57 GMT
Find Nodes?
    Voting Booth?
    Which type of climate do you prefer to live in?

    Results (74 votes). Check out past polls.