Best way to handle/search CSV file in Perl?

Speedfreak has asked for the wisdom of the Perl Monks concerning the following question:

Hej All,

Interesting one for you. I have a CSV file which I need to search through. I dont want to use DBI::CSV because its a hassle, just want to work with filehandles and arrays.

Here comes the problem, the array would be 6000x7 in dimension!

What I need to do is read each line into an array and then perform a soundex on a couple of fields and put those that match into another array.

Gut feeling tells me that going DBI::CSV would be inefficient because of the overheads of creating connections and record sets and besides, just reading and splitting the file at the moment is only taking a few seconds.

Would I be better reading the input file line by line and putting those that match into a second array?

- Jed

Comment on Best way to handle/search CSV file in Perl?

Replies are listed 'Best First'.
Re: Best way to handle/search CSV file in Perl? by lhoward (Vicar) on May 21, 2000 at 06:17 UTC
If your program needs to scan the data multiple times each time you run it and you have the memory to spare to hold the whole file in memory (almost certainly w/ a file as small as 6000x7) then by all means read the file in once and store it in an array (or other data structure) and process off of that. If you are only going to need to scan the data once each time your program is run then you will see no over-all preformance gains by pre-loading it into memory. One module you may want to check out is Text::CSV. It provides parsing for CSV formatted files.	[reply]
Re: Best way to handle/search CSV file in Perl? by ZZamboni (Curate) on May 21, 2000 at 04:34 UTC
If each line can be processed separately, I would definitely read line by line. This will make it easier for your program to handle if the input size grows in the future. --ZZamboni	[reply]
RE: Re: Best way to handle/search CSV file in Perl? by takshaka (Friar) on May 21, 2000 at 05:42 UTC
Indeed. Why do you want to read the whole file into an array? That'll eat up gobs of memory if the file is large (of course, if the file is very large, you should probably consider alternatives to flat-file storage).	[reply]
Re: Best way to handle/search CSV file in Perl? by Speedfreak (Sexton) on May 21, 2000 at 16:16 UTC
Thanks all, After some testing and trials it does turn out to be more efficient to just process each line as it comes in. - Jed	[reply]
Re^2: Best way to handle/search CSV file in Perl? by Anonymous Monk on Nov 23, 2004 at 03:01 UTC
This is eaxctly what I need to do. I am new to PERL so having some code for me to base my stuff on would be a great help. If you still have this piece of code would you mind posting it? Thanks	[reply]
Re: Best way to handle/search CSV file in Perl? by princepawn (Parson) on May 21, 2000 at 04:49 UTC
There is an article called "Prequel to SQL" in an old issue of the Perl Journal. I can give you an exact ref monday. But if you are a subscriber, you can d/l the article yourself today.	[reply]