Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re^2: Lost in compressed encodings

by Skeeve (Parson)
on Apr 06, 2020 at 08:53 UTC ( #11115123=note: print w/replies, xml ) Need Help??

in reply to Re: Lost in compressed encodings
in thread Lost in compressed encodings

Thanks Corion. I already had the feeling that the sequence somehow is the issue.

Unfortunately providing binmode after IO::Uncompress did not help.

My changed code:

open my $in, '<:raw', $filename or die "Can't read $filename: $!\n +"; if ($filename=~/\.gz$/) { $in= new IO::Uncompress::Gunzip $in, { AutoClose => 1 }; } binmode $in, ':utf8';

It still works with uncompressed and not with compressed data.

Seems I will have to manually decode each line…


Replies are listed 'Best First'.
Re^3: Lost in compressed encodings
by Corion (Patriarch) on Apr 06, 2020 at 08:59 UTC

    I think that IO::Uncompress::Gunzip only understands ->binmode() and not ->binmode(':utf8');. The documentation (now that I read it ...) even says:

    This is a noop provided for completeness.

    If you are able to install PerlIO::gzip, that one should work with stacking other decoding mechanisms on top of it.

    If you have a gzip binary available, you can use that to decompress:

    my $in; if( $filename =~ /\.gz$/ ) { open $fh, "gzip -cd "$filename" |' or die "Can't read from gzip $filename: $!/$?"; } else { open $in, '<:raw', $filename or die "Can't read $filename: $!\n"; }; binmode $fh, ':utf8';

      Thanks for all your suggestions, Corion.

      As my usecase is a module which reads a (kind of) CSV file, there are just 2 places, where I read a line from the file.

      So I decied to do the "manual" decoding:

      open my $in, '<:raw', $filename or die "Can't read $filename: $!\n"; if ($filename=~/\.gz$/) { $in= new IO::Uncompress::Gunzip $in, { AutoClose => 1 }; } # later on $_= <$in>; chomp; @headers= split /\t/, lc decode('UTF-8' => $_); # That's not really required as the header will always be # ASCII-characters… But for completeness sake… # further down I have a loop while (<$in>) { chomp; # … @line{@headers}= split /\t/, decode('UTF-8' => $_);


        I would suggest using Text::CSV_XS to read/parse the CSV. It already knows how to deal with UTF-8.

        It is capable of using a TAB as separation character:

        use Text::CSV_XS; my $csv = Text::CSV_XS->new ({ binary => 1, sep_char => "\t", auto_dia +g => 1 }); my @headers = $csv->header ($in); # Read the docs, there are options p +ossible here while (my $row = $csv->getline ($in)) { # ... }

        update: I realized later that Text::CSV_XS' csv function already supports gzip as part of the encoding attribute.

        use Text::CSV_XS qw( csv ); use PerlIO::via::gzip; my $aoa = csv (in => "test.csv.gz", encoding => ":via(gzip)");

        Enjoy, Have FUN! H.Merijn

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11115123]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (1)
As of 2022-12-06 20:06 GMT
Find Nodes?
    Voting Booth?

    No recent polls found