in reply to Re: Perl script for the post processing of one CSV file
in thread Perl script for the post processing of one CSV file

You can stream it and prevent memory hogs on big data. Additionally, install Text::CSV_XS for speed

use Text::CSV_XS "csv";

my %sums; # Column sums. $sum{column} += $value
my @head;
my @pats;

csv (in => "input.csv", out => undef, bom => 1, kh => \@head, on_in =>
+ sub {

    my ($csv, $row) = @_;

    unless (@pats) {
        # Fetch the "patN" column names, in order. This works
        # for single digit "patN" names, as that was your
        # example. Multi-digit names will require a more complex
        # sort, left as an exercise to the reader.
        # XXX @pats = sort { grep m/^pat\d+$/ } @head;
        @pats = sort grep m/^pat\d+$/ => @head; # This line fixed
        }

    for (@{$row}{@pats}) {
        # You are now iterating over every patN value,
        # in order. Perform your transformation
        }

    # Just an example.
    $sums{$_} += $row->{$_} for @pats;
    });

[download]

update: I changed the grep line which I blindly copied from the original code


Enjoy, Have FUN! H.Merijn

Replies are listed 'Best First'.
Re^3: Perl script for the post processing of one CSV file
by rjt (Curate) on Oct 03, 2019 at 07:59 UTC
    You can stream it and prevent memory hogs on big data. Additionally, install Text::CSV_XS for speed use Text::CSV_XS "csv";

    Install Text::CSV_XS, yes, but don't explicitly use Text::CSV_XS. Text::CSV is smart enough to pull in the XS version if installed, and will fall back to pure Perl if not. There is usually no point in having the script break if XS isn't available.

    perl -MText::CSV -e 'print Text::CSV->module'
Text::CSV_XS

    [download]

    I agree with your streaming suggestion. I had opted to keep my example simple given the 20k line input. Good to teach the streaming approach, though. ++

    use strict; use warnings; omitted for brevity.
[reply]
[d/l]
[select]
Re^3: Perl script for the post processing of one CSV file
by kshitij (Sexton) on Oct 03, 2019 at 07:22 UTC
    Hi Superdoc ,

    Thanks , I have tried it but getting an error as stated below

    Not enough arguments for grep at csv_to_output_report.pl line 18, near "m/^pat\d+$/ }"

    Could you guide me to resolve this error ?

    Thanks Kshitij

[reply]
Re^3: Perl script for the post processing of one CSV file
by kshitij (Sexton) on Oct 03, 2019 at 07:59 UTC

    Hi Superdoc ,

    I cant use these subroutines "bom" and "kh" since I am using older version of perl. Can you help me out with the code without using these subroutines ?

    Thanks Kshitij

[reply]

      Those are arguments, not subroutines, and they are part of Text::CSV, not Perl itself. What version of Perl are you using (perl -v from a prompt), and what version of Text::CSV do you have installed? Find that out with perl -MText::CSV -e 'print Text::CSV->VERSION' again from a prompt. The latest is 2.00, released in May of this year.

      You can upgrade your version of Text::CSV using CPAN (and install Text::CSV_XS while you're at it!), and you will be able to use these attributes. Using CPAN

[reply]
[d/l]
[select]
[reply]
      since I am using older version of perl

      I have some great news for you. You can upgrade individual modules without upgrading the entire perl installation. However, if your perl is so old (which version?) why not take this opportunity to upgrade the entire system anyway?

[reply]

In Section Seekers of Perl Wisdom