Re^2: agregating columns in several csv files

Hi Corion, thanks for the quick reply, however
MaxFDLimit ~= 65K
ulimit -n 4096
So I'm not hitting the OS FD limit, was hitting my shells limit until I adjusted ulimit -n.

If it were in a database this would be easier, however I'm restricted in the tools to hand for this,(Can't take data off the server). It's also worth noting that I can only use core modules as i can't install anything on the server :(

Comment on Re^2: agregating columns in several csv files

Replies are listed 'Best First'.
Re^3: agregating columns in several csv files by Corion (Patriarch) on Jun 05, 2009 at 08:03 UTC
Anonymous Monk has found why the diamond operator does not work. DB_File or any other BTree storage likely is installed already with your Perl, like DBM_File or SDBM_File, and they can be used to conveniently rearrange the data and adress it by a common key. Have you considered concatenating all your CSV files into a single file? If you then sort that single file by the columns you want to use as keys, all your calculations become much simpler as you only have one set of variables that you purge whenever your key changes. I will mention Yes, even you can use CPAN, but it's only for future reference as DBD::SQLite will need a C compiler, so you can't completely paste it into your script.	[reply]
Re^3: agregating columns in several csv files by Anonymous Monk on Jun 05, 2009 at 08:04 UTC
Yes, even you can use CPAN. You can also inline module Text::CSV_PP using Inline::Module, and you get all the benefits of an actively maintained module, with good kwalitee.	[reply]