comment on

That was easy enough. Thanks! Of course it still doesn't work because some characters don't appear to be UTF-8.

I wrote this code to try to figure out what encoding it is, but for any file with the "special" characters I just get the "Didn't work" message.

#!/usr/bin/perl
use strict;
use warnings;
use Encode::Guess;

undef $/;  # slurp on

my $dir = '.';
if (@ARGV > 0) {
  $dir = $ARGV[0];
}

opendir DIR, $dir or die "Can't opendir '.': $!\n";
my @files = grep /\.csv$/i, readdir(DIR);
closedir DIR;

Encode::Guess->add_suspects(qw(latin1 cp1252)); # What else?
foreach my $file (@files) {
  open my $fh, "<:raw", "$dir/$file"
    or die "Can't open $!\n";
  my $data = <$fh>;
  close $fh;

  my $enc = guess_encoding($data);
  if (ref $enc) {
    print "$file: " . $enc->name . "\n";
  } else {
    print "Didn't work for: $file\n";
  }
}

exit;
[download]

This file was generated on Windows by exporting from Outlook. The Windows is setup for American English, but the keyboard is Danish. :-/ All the files that DO work are reported with "ascii" encoding (as I expect).

In reply to Re^2: Reading CSV Files Containing UTF8 Characters by shoness
in thread Reading CSV Files Containing UTF8 Characters by shoness

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


go ahead... be a heretic
	PerlMonks