Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??
I have a file that *should* be all ISO-latin, but the program that created it seems to sprinkle UTF-8 characters round in it.

If that's the case, then that program is horribly broken and I would recommend trying to see what you can do to fix that. Anyway, do you have any sample data that shows both UTF-8 and Latin-1 data?

EF BB BF is the Byte order mark encoded as UTF-8, it can be present at the beginning of UTF-8 encoded files. The fact that in your second sample it appears after a series of commas could mean that the program is trying to write a CSV file and used a UTF-8 encode function that adds the BOM on individual fields, or it slurped a file with the wrong encoding and used that as the contents of the field. If this guess is correct, then maybe a solution would be to first parse the CSV file and then individually decode the fields with different encodings, though I would consider that a pretty ugly workaround, plus you'd have to know the encodings (or guess them, which is a workaround in itself).

Can you tell us more about this program, the file format it is outputting, and give more example data that shows the problem?

Update just to address the question in the title and node: simply clobbering any non-ASCII characters without understanding the input data is almost never the right solution, because you'll almost certainly also delete important characters. Instead, first fix the encoding problems, and if you then really want to ASCIIfy your (correctly decoded!) Unicode data, you can use e.g. Text::Unidecode.


In reply to Re: getting rid of UTF-8 by haukex
in thread getting rid of UTF-8 by BernieC

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (6)
As of 2023-03-28 09:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Which type of climate do you prefer to live in?






    Results (66 votes). Check out past polls.

    Notices?