Re^4: Text CSV_XS memory crash

Hmm, I'm starting to see the light. I only need to work on one column at a time, read in the column, select min/max and distinct, then load the next column and repeat.

I'll go back to the drawing board, I think I'm pretty close...

Comment on Re^4: Text CSV_XS memory crash

Replies are listed 'Best First'.
Re^5: Text CSV_XS memory crash by Anonyrnous Monk (Hermit) on Feb 02, 2011 at 16:18 UTC
select min/max and distinct FWIW, things like min/max (and many other statistics) can also be computed sequentially (or incrementally), i.e. without storing all values in memory prior to starting the computations. Maybe that's worth reconsidering...	[reply]
Re^5: Text CSV_XS memory crash by marto (Cardinal) on Feb 02, 2011 at 16:01 UTC
Have you considered using DBD::CSV, or alternativly importing your data into a database of some sort?	[reply]
Re^6: Text CSV_XS memory crash by Tux (Canon) on Feb 02, 2011 at 18:18 UTC
Using DBD::CSV will never be a valid option when reading a CSV file in full will cause memory hogs, as DBD::CSV will read the whole file into memory always. Nothing to prevent that. That means that the memory consumption for CSV files accessed with DBI might be up to double the size of the complete dataset as returned bi `Text::CSV_XS->getline_all ()`. Enjoy, Have FUN! H.Merijn	[reply] [d/l]