Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^3: Perl detect utf8, iso-8859-1 encoding

by haj (Curate)
on Jul 25, 2020 at 08:50 UTC ( #11119791=note: print w/replies, xml ) Need Help??


in reply to Re^2: Perl detect utf8, iso-8859-1 encoding
in thread Perl detect utf8, iso-8859-1 encoding

To check whether data are valid UTF-8 is rather straightforward. Here's the example, slightly modified from the synopsis of Encode:
use Encode qw(decode encode); $characters = decode('UTF-8', $octets, Encode::FB_CROAK | Encode::LEAVE_SRC);

This code will die if there are invalid data, so you would wrap it into the exception handler of your choice, plain eval and Try::Tiny seem to be popular.

BTW: as jcb already indicated, chances are excellent that if data pass as UTF-8, they actually are UTF-8. All bytes of multibyte characters in valid UTF-8 strings are in the range \x80 to \xFF, and in particular the bytes 2-4 are in the range \x80-\xBF. You just can't build readable text from characters in that range in any of the ISO-8859-* encodings, and about half of that range are "unprintable" control characters from ISO/IEC 6429.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11119791]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (6)
As of 2021-01-19 19:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?