Re^2: Perl script for the post processing of one CSV file

You can stream it and prevent memory hogs on big data. Additionally, install Text::CSV_XS for speed

use Text::CSV_XS "csv";

my %sums; # Column sums. $sum{column} += $value
my @head;
my @pats;

csv (in => "input.csv", out => undef, bom => 1, kh => \@head, on_in =>
+ sub {

    my ($csv, $row) = @_;

    unless (@pats) {
        # Fetch the "patN" column names, in order. This works
        # for single digit "patN" names, as that was your
        # example. Multi-digit names will require a more complex
        # sort, left as an exercise to the reader.
        # XXX @pats = sort { grep m/^pat\d+$/ } @head;
        @pats = sort grep m/^pat\d+$/ => @head; # This line fixed
        }

    for (@{$row}{@pats}) {
        # You are now iterating over every patN value,
        # in order. Perform your transformation
        }

    # Just an example.
    $sums{$_} += $row->{$_} for @pats;
    });
[download]

update: I changed the grep line which I blindly copied from the original code

Enjoy, Have FUN! H.Merijn

Comment on Re^2: Perl script for the post processing of one CSV file Download Code

Replies are listed 'Best First'.
Re^3: Perl script for the post processing of one CSV file by rjt (Curate) on Oct 03, 2019 at 07:59 UTC
You can stream it and prevent memory hogs on big data. Additionally, install Text::CSV_XS for speed `use Text::CSV_XS "csv";` Install Text::CSV_XS, yes, but don't explicitly `use Text::CSV_XS`. Text::CSV is smart enough to pull in the XS version if installed, and will fall back to pure Perl if not. There is usually no point in having the script break if XS isn't available. `perl -MText::CSV -e 'print Text::CSV->module' Text::CSV_XS` [download] I agree with your streaming suggestion. I had opted to keep my example simple given the 20k line input. Good to teach the streaming approach, though. ++ `use strict; use warnings;` omitted for brevity.	[reply] [d/l] [select]
Re^3: Perl script for the post processing of one CSV file by kshitij (Sexton) on Oct 03, 2019 at 07:22 UTC
Hi Superdoc , Thanks , I have tried it but getting an error as stated below Not enough arguments for grep at csv_to_output_report.pl line 18, near "m/^pat\d+$/ }" Could you guide me to resolve this error ? Thanks Kshitij	[reply]
Re^3: Perl script for the post processing of one CSV file by kshitij (Sexton) on Oct 03, 2019 at 07:59 UTC
Hi Superdoc , I cant use these subroutines "bom" and "kh" since I am using older version of perl. Can you help me out with the code without using these subroutines ? Thanks Kshitij	[reply]
Re^4: Perl script for the post processing of one CSV file by rjt (Curate) on Oct 03, 2019 at 08:07 UTC
Those are arguments, not subroutines, and they are part of Text::CSV, not Perl itself. What version of Perl are you using (`perl -v` from a prompt), and what version of Text::CSV do you have installed? Find that out with `perl -MText::CSV -e 'print Text::CSV->VERSION'` again from a prompt. The latest is `2.00`, released in May of this year. You can upgrade your version of Text::CSV using CPAN (and install Text::CSV_XS while you're at it!), and you will be able to use these attributes. Using CPAN	[reply] [d/l] [select]
Re^5: Perl script for the post processing of one CSV file by kshitij (Sexton) on Oct 03, 2019 at 08:15 UTC
Please find the output 1.32kulshrk@idc-cad-vnc05 atpg % perl -MText::CSV -e 'print Text::CSV->VERSION' 1.32kulshrk@idc-cad-vnc05 atpg %	[reply]
Re^6: Perl script for the post processing of one CSV file by rjt (Curate) on Oct 03, 2019 at 09:00 UTC
Re^4: Perl script for the post processing of one CSV file by hippo (Bishop) on Oct 03, 2019 at 08:12 UTC
since I am using older version of perl I have some great news for you. You can upgrade individual modules without upgrading the entire perl installation. However, if your perl is so old (which version?) why not take this opportunity to upgrade the entire system anyway?	[reply]


Think about Loose Coupling
	PerlMonks