I have modified your code to process the incoming stream looking for min, max and unique values. Configure @min_, @max_ and @unique_ columns to control which fields are processed.

#!/usr/bin/perl use strict; use warnings; use Text::CSV_XS; use Data::Dumper; my $file = "./test.csv"; my @min_columns = qw( ozone ozone_f qa_code ozone_8hr); my @max_columns = qw( ozone ozone_f qa_code rank ); my @unique_columns = qw( site_id ); my $data; my $counter; my $csv = Text::CSV_XS->new({ binary => 1, auto_diag => 1, }) or die "Cannot use CSV: " . Text::CSV_XS->error_diag(); open my $FH2, "<:encoding(utf8)", $file or die "$file: $!"; my $columns = $csv->getline($FH2); for (@$columns) {s/^\s+//; s/\s+$//}; $csv->column_names(@$columns); while (my $row = $csv->getline_hr($FH2)) { process_row($row); } print "done with loop after $counter rows\n"; $csv->eof or $csv->error_diag(); close $FH2; print Dumper $data; sub process_row { my $row = shift; $counter++; print "processing: $counter\n" if ($counter % 100 == 0); for my $column (@min_columns) { my $val = $row->{$column}; my $d = $data->{$column}{min}; my $update = {value => $val, row => $row}; $data->{$column}{min} = !exists $d->{value} ? $update : $d->{value} > $val ? $update : $d; } for my $column (@max_columns) { my $val = $row->{$column}; my $d = $data->{$column}{max}; my $update = {value => $val, row => $row}; $data->{$column}{max} = !exists $d->{value} ? $update : $d->{value} > $val ? $update : $d; } for my $column (@unique_columns) { my $val = $row->{$column}; $data->{$column}{unique}{$val}++; #could just be =1, but now we can get unique and count distinc +t } }

I switched to Text::CSV_XS from _PP, as the example data had an issue around line 296. (PP parse error 2025, loose escape char)

This is an excellent spot to use DataCube, but that module is gone now. I would like David to fix it and put it back on CPAN. backpan DataCube

If you're going to be doing ad-hoc analysis of csv files, using r language may prove useful.

Edit: added updates from Tux's comment


In reply to Re: Text CSV_XS memory crash by spazm
in thread Text CSV_XS memory crash by glepore70

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.