Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

Hi Ken!

Tried to find a general solution to the problem reported in Re: uparse - Parse Unicode strings.

Short explanation of the problem:
There are two basic ways to get correct UNICODE input from the elements in @ARGV:

  • implicit decoding with a runtime option -CA or an environment setting PERL_UNICODE=A
  • explicit decoding using Encode::decode
Either may be used, but not both.

A script that expects UNICODE data from @ARGV cannot easily detect if the implicit decoding is in effect, especially because -CAL makes the behaviour locale-dependent.

The best solution I could find is to check if the data in question is already marked to be in UTF-8. Encode::is_utf8 (or the equivalent utf8::is_utf8) may be used to check this flag, which results in a small modification to your script:

diff --git a/uparse b/uparse index f5edb92..b05e12a 100755 --- a/uparse +++ b/uparse @@ -23,11 +23,11 @@ use constant { NO_PRINT => "\N{REPLACEMENT CHARACTER}", }; -use Encode 'decode'; +use Encode qw(decode is_utf8); use Unicode::UCD 'charinfo'; for my $raw_str (@ARGV) { - my $str = decode('UTF-8', $raw_str); + my $str = is_utf8($raw_str) ? $raw_str : decode('UTF-8', $raw_str +); print "\n", SEP1; print "String: '$str'\n"; print SEP1;

What do you think about this?



In reply to Decoding @ARGV [Was: uparse - Parse Unicode strings] by jo37
in thread uparse - Parse Unicode strings by kcott

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?

What's my password?
Create A New User
Domain Nodelet?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (6)
As of 2024-04-24 06:51 GMT
Find Nodes?
    Voting Booth?

    No recent polls found