Re: Handling japanese file & comparison

Do you know what character encoding was used to write the "Japanese compressed file"? Is it Unicode (utf8, utf16)? (Shift) JIS? EUC? Something else?

If you don't know how to answer that question, your first task is to figure out how to answer that question. If you don't have some reliable reference or documentation for your data that answers the question, use Encode::Guess for that. Try it, and if you can't figure it out, show us what you tried.

As for the rest, it sounds like you want to read file B first, store its "rid" values as hash keys (and if there's anything else you need from B in your output, store that as hash values).

Then read each A file in turn, and output whatever you want when there's a suitable match to one of the hash keys from file B. Your description is not clear, but I gather that the logic would be something like:

use strict;

my %B_rids;

# ... open file B, read it to fill %B_rids hash, then: 

my @A_files = ... # do something to get list of A file paths/names

for my $A_file ( @A_files ) {
    unless ( open( my $afh, "<", $A_file )) {
              # update: you'll add encoding on 2nd arg, e.g. "<:utf8"
        warn "Open failed for $A_file: $!\n";
        next;
    } 
    while (<$afh>) {
        chomp;
        my @fields = split /\t/;
        if ( exists( $B_rids{ $fields[4] } )) { # does rid1 exist in B
+?
            # do something
        }
        elsif ( exists( $B_rids{ $fields[5] } )) { # does rid2 exist i
+n B?
            # do something else
        }
    }
}
[download]

(updated to fix typo in list of encodings)

Comment on Re: Handling japanese file & comparison Download Code

Replies are listed 'Best First'.
Re^2: Handling japanese file & comparison by CSharma (Sexton) on Oct 04, 2015 at 12:34 UTC
Thanks a lot graff for the help! 1. file encoding is : utf-8 2. How to open zipped file with encoding? I've written my code, review it & let me know my mistakes. 1. I've read cat file into %catHash 2. opened the main file from which I've to extract line if exists in cat file. 3. if $line17 doesn't have value, $line14 value to be used; if both don't have, then read next line. Am I right below? 4. $line17 & $line14 in main file, some have leading 0's but not in that value in cat file. for example: cat file value: 101010100 & main file has: 0101010100 What do you suggest to handle this situation. I've tried replacing these leading 0's. I'm new to perl & programming both, please help. #!/usr/bin/perl use strict; my $f = "vc_20151003_1.zip"; my $cat = "5426085.csv"; my $file = $f; my $file =~ s/.zip/.csv/; ## Read the cat file into hash %catHash = (); open(CAT, $cat) \|\| die "$cat couldn't be opened\n"; while(<CAT>) { @line = split /,/; $catHash{$line[1]}++; } close(CAT); open(OUT, ">".$file) \|\| "$file couln't be opened for write\n"; open(IN,sprintf("zcat %s \|", $f)) \|\| die "Could not open pipe for $f +: $!"; while(<IN>) { chomp; my @line = split /\t/; $line[17] =~ s/^0+//; $line[14] =~ s/^0+//; if(exists($catHash{$line[17]}) \|\| exists($catHash{$line[14]})) + { print OUT join("\t", @line) . "\n"; } else { next; } } close(IN); close(OUT); [download]	[reply] [d/l]
Re^3: Handling japanese file & comparison by CSharma (Sexton) on Oct 05, 2015 at 13:39 UTC
Well, I've done this myself. Thanks for help! ~Csharma	[reply]


Come for the quick hacks, stay for the epiphanies.
	PerlMonks