Re: Efficiency issue when reading large CSV files

Is there a reason you can't simply split based off the comma?

@line = split(/\,/, $_);
[download]

I don't understand, from your post, where the need for an external library comes into the picture.

Comment on Re: Efficiency issue when reading large CSV files Download Code

Replies are listed 'Best First'.
Re^2: Efficiency issue when reading large CSV files by Takuan Soho (Acolyte) on Jun 25, 2009 at 18:45 UTC
Indeed, I would prefer not to use an external library. The problem is that the CSV format allows using a coma inside quotes. It is unlikely that there will be a quoted coma in my input... but "unlikely" does not mean that there will never be such a thing. If I split on coma and there are quoted comas, the computation could be totally messed up, but in ways that would make the mess almost unnoticeable.	[reply]
Re^3: Efficiency issue when reading large CSV files by jrsimmon (Hermit) on Jun 25, 2009 at 19:18 UTC
I suspect you could still handle this without an external library. Complex regex aren't my forte, but I know this forum where that seems to be a pretty popular pasttime... Something along the lines of: `$_ =~ s/some expression meaning a quoted comma/some placeholder/g; @line = split(/\,/, $_); #if the commas are needed foreach my $line (@line){ $line =~ s/some placeholder/,/g; }` [download] or `@line = split(/some expression that only matches commas not within quo +tes/, $_);` [download] You're going to pay a bit of a performance penalty on the 2nd substitution or the more complex matching statement, but it's linear not exponential, so not too bad. Given what you've seen with Text::CSV, I'd wager it'll still be an improvement. Perhaps I'm a purist, but I am loathe to pull in another module when the task can be accomplished without it.	[reply] [d/l] [select]