go ahead... be a heretic | |
PerlMonks |
comment on |
( [id://3333]=superdoc: print w/replies, xml ) | Need Help?? |
(OP here -- sorry, I should have obtained a username before starting this) Sorry also for the delay, I stopped to do more tests based on the useful information you've all given me. The "wide character" error happens if I'm trying to decode a string that already has what I have been calling "unicode" extended characters in it. (Remember I said this is all a great mystery to me and I'd really like it to all go away forever? That includes my incorrect terminology.) That is, if it's already got characters such as \x{103}, trying to decode them will produce that error. This turns out to be because one of my data sources sends extended characters in one format and one in a different format (this is an API that has to merge data from several sources for a single output stream). Or, more concretely: One of my data sources sends lower-case-a-with-breve ă as \xc4\x83, which is the kind that does need translating for my purposes, and the other data source sends it as \x{103}, which for my purposes is already translated into the format I need. decode('UTF-8') works properly on the former and errors on the latter, which seems to be correct behavior based on what you've said. I didn't realize the two data sources were doing it differently (neither of them has any documentation of what they do, alas) and I picked the wrong horse for my previous test. The reason I was calling the former of those "UTF-8" and the latter "Unicode" was because of pages like https://www.utf8-chartable.de/unicode-utf8-table.pl?start=256&names=-&utf8=string-literal, where I look up characters for when I have to translate them by brute force ... I'm still not sure of the correct term for the longer Unicode encoding where ă is \x{103} (AKA U+0103). Anyway, thank you! It does look like decode() does the right thing, when its user isn't dumb. In reply to Re^4: UTF-8 and Unicode the hard way
by Anonymous Monk
|
|