I am trying to parse the content of car.csv and place it into a directory neatly. My code works perfectly, its just the performance is lagging since our car database has over 5 million lines. I suspect the bottleneck is in the open() and close() calls.
$ cat car.csv make,model,color,year Honda,Civic,Red,2008 Toyota,Camry,Blue,2002 Honda,Accord,Red,1992 Nissan,Sentra,Blue,2009 Ford,Focus,Green,2009 Honda,Civic,Red,2003 Toyota,Corolla,Green,2002 Honda,Civic,Red,1992 Honda,Civic,Green,2008 Toyota,Camry,Orange,2002 Honda,Accord,Black,1992 Nissan,Sentra,White,2009 Ford,Focus,Green,2007
#!/usr/bin/perl -w use strict; #Run the script like, tail -n +2 car.csv | ./foo.pl #I want to ignore the header my $make; #Car Make my $model; #Car Model my $out_dir="/var/tmp/cars"; #Output directory for while (<>) { ($make,$model) = split(/,/,$_,4); system("mkdir -p $out_dir/$make"); open FILE, ">>$out_dir/$make/$model" or die $!; print FILE $_; close FILE; }
The output will be:
$ cat /var/tmp/cars/Honda/Civic Honda,Civic,Red,2008 Honda,Civic,Red,2003 Honda,Civic,Red,1992 Honda,Civic,Green,2008 $ cat /var/tmp/cars/Honda/Accord Honda,Accord,Red,1992 Honda,Accord,Black,1992 $ cat /var/tmp/cars/Ford/Focus Ford,Focus,Green,2009 Ford,Focus,Green,2007
Any thoughts? TIA

In reply to handler performance question by magawake

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.