Hello, To do some bioinformatics researches, I have to read fields from huge files and do some basic operations on those fields. As long as I work with tab separated files, everything works fine. I can split the lines using something like : "@line = s/\t/, $_;". I can access the fields and do a simple printing task like this:print "$line2\t$line[0]"; (Of course, the manipulations are a little more complex, but this is for the sake of testing the issue). When I started getting CSV files, and tried to do the same thing using Text::CSV, everything jammed. Here is a simple program that illustrates what I did, and record the time it took: 

### Test program

use Text::CSV;

$input_file = "ACGT.csv";
$output_file = "test.txt";

if (! open ENTREE, "<", $input_file) {
   print "Could not open handle ENTREE for $input_file. $!\n";}
if (! open SORTIE, ">", $output_file) {<br>
   print "Could not open handle SORTIE for $output_file. $!\n";}

$start = time();

my $csv_reader = Text::CSV->new();
my @columns;

while (<ENTREE>) {
   $csv_reader->parse($_);
   @columns = $csv_reader->fields();
   print SORTIE "$columns[2]\t$columns[0]\n";
}

$end = time();
$duration = $end - $start;

print "Reading took $duration seconds.\n";

close ENTREE;
close SORTIE;

[download]

On a huge file (a couple hundreds of columns, around a million rows), it took 3858 seconds. More than an hour to just go through a file!!! Am I doing something inherently wrong?

As a comparison, I transformed the CSV file in a tab separated file using a small python script based on the python CSV module, and it took less than 200 seconds. Then, back to the perl script, I did the same manipulation (using the split function this time) in 273 seconds.

Do you know of a way to deal with CSV files in perl that will allow me to get to the same kind of efficiency ... without having to use a python script on the side?

Thank you very much!

In reply to Efficiency issue when reading large CSV files by Takuan Soho

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.