Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Re: How to convert a string like this: "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\321\200\320\260" to a proper utf8?

by Ovid (Cardinal)
on Apr 28, 2016 at 06:04 UTC ( #1161729=note: print w/replies, xml ) Need Help??

in reply to How to convert a string like this: "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\321\200\320\260" to a proper utf8?

Take a look at Encode::DoubleEncodedUTF8. The following returns your original string:

#!/usr/bin/env perl use 5.18.0; use warnings; use utf8::all; use Encode; use Encode::DoubleEncodedUTF8; my $bytes = "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\ +321\200\320\260"; my $fixed = decode( "utf-8-de", $bytes ); say $fixed;

Update: A strong word of caution about the above. Fixing "broken" Unicode data is fraught with error, particularly if you can't guarantee the source of the error. If this bad data is coming from outside your program, then try to contact whoever generates that bad data and see if you can get it fixed there. Otherwise, you may have to trust yourself to be lucky on this.

However, if the bad data is being generated from within your program, try to fix the underlying bug before you try to "fix" the bad strings in question. It will save you much grief in the long run.

  • Comment on Re: How to convert a string like this: "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\321\200\320\260" to a proper utf8?
  • Download Code

Replies are listed 'Best First'.
Re^2: How to convert a string like this: "\321\201\321\202\321\200\321\203\320\272\321\202\321\203\321\200\320\260" to a proper utf8?
by igoryonya (Pilgrim) on Apr 28, 2016 at 11:41 UTC
    The output comes from 'git status' command.
    I have some filenames in Russian in my project folder.
    The surprising thing is that git shows all other Russian text correctly, only file names appear encoded.
    I've already seek for answers on git's irc chanel. Nobody could help me there.
    Searched on the internet, but the closest thing that I've found about this issue, is a 2 year old similar to mine question, unresolved.
    So, I thought, that, since the git's community can't help me, I will just write an output reformatter. Something, like this:
    git status -s|perl -nle 's/\\([[:digit:]]{3})/sprintf "%c", oct($1)/ge +; print'
    After seeing, that it works, I've moved perl oneliner to a file, so, now I do:
    git status -s|
      I believe the quoting can be disabled with core.quotepath option. See man git-config for details.

      Also, that reminds me of... :)

        I thought, that I did 'git config --global core.quotepath false', but it didn't change anything, then, I started trying to figure out how to do it in perl.

        Now, since you've wrote about it, I went to 'git config --global -l' to get a copy of that parameter for a proof and it wasn't there.
        How could it be, I remember, that I did it. I've looked at the command history and found 'git config --global --get core.quotepath false'. :), so I've forgot to remove --get parameter from the previous git command, when typing this one.

        At first, I thought, I've wasted so much time, because of this little mistake, but then, I've figured, no, this escaped string has bugged me before and, finally, I've found a solution to it, so, I was diverted from the original problem, but didn't waste any time.

        P.S.: "Also, that reminds me of... :)"
        ...Yes, this unicode thing still haunts me. I still can't tackle it completely. There is so much documentation on it. I read it occasionally, but can't read it all in one, or even 10 seats :). Also, much of the stuff gets forgotten over time, but little by little, I refine my unicode knowledge. That post helped me to understand, that not everything can be unicode. Files have to be bytes internally, only convert them to unicode, when outputting to the screen.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1161729]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (2)
As of 2023-03-31 03:27 GMT
Find Nodes?
    Voting Booth?
    Which type of climate do you prefer to live in?

    Results (74 votes). Check out past polls.