Re: Text CSV_XS memory crash

Nearly the exact same data is available at:

http://views.cira.colostate.edu/documents/Data/SourceFiles/EPA%20Clean%20Air%20Status%20and%20Trends%20Network%20%28CASTNet%29/CASTNet%20Ozone/

ozone_2007.csv

Comma separated, quote enclosed.

That's a 50MB document, just cat it to itself to get a 100MB sample file that crashes for me.

pmvers Text::CSV Text::CSV_XS Text::CSV_PP
Text::CSV: 1.21
Text::CSV_XS: 0.80
Text::CSV_PP: 1.29
[download]

Thanks.

Comment on Re: Text CSV_XS memory crash Download Code

Replies are listed 'Best First'.
Re^2: Text CSV_XS memory crash by Anonymous Monk on Feb 02, 2011 at 15:16 UTC
50mb file is still a 50mb file ;) 1-10-20 lines of generator is a ton smaller, and fits on perlmonks without compression :)	[reply]
Re^2: Text CSV_XS memory crash by Tux (Canon) on Feb 02, 2011 at 18:09 UTC
$ csv-check ozone_2007.csv Checked ozone_2007.csv with csv-check 1.5 using Text::CSV_XS 0.80 OK: rows: 756689, columns: 8 sep = <,>, quo = <">, bin = <0>, eol = <"\r\n"> $ perl ozone.pl ozone_2007.csv Reading ozone_2007.csv with Text::CSV_XS-0.80 ... Data size = 4194400, total size = 409046824 $ cat ozone.pl #!/pro/bin/perl use strict; use warnings; use autodie; use Text::CSV_XS; use Devel::Size qw( size total_size ); my $file = shift; print "Reading $file with Text::CSV_XS-$Text::CSV_XS::VERSION ...\n"; my $csv = Text::CSV_XS->new ({ binary => 1, auto_diag => 1 }); open my $fh, "<:encoding(utf-8)", $file; my $dta = $csv->getline_all ($fh); printf "Data size = %d, total size = %d\n", size ($dta), total_size ($ +dta); $ [download] or in a one-liner $ perl -MDevel::Size -MText::CSV_XS -wle'print Devel::Size::total_size +(Text::CSV_XS->new()->getline_all(*ARGV))' ozone_2007.csv 409046824 $ [download] Enjoy, Have FUN! H.Merijn	[reply] [d/l] [select]