While I agree with the other posters that for large datasets and cases where speed is critical, you may be better off with one of the "real" databases, but to say it's DBD::CSV and therefoe "there is little place for improvements" is much too broad a statement. Yes DBD::CSV can be slow, but that doesn't mean there is never a way to optimize. For example I suspect that your code will run much faster if you replace all of your prepares and executes and loops and file opening and file writing with this one statement, it will (if I've understood what you're doing) create a new table with the join of the rows from the other table three tables. 
$dbh->do("
   CREATE TABLE trafficstats
      AS SELECT key, sec.id, sec.hid, 
                name,state,ccs,kbps,mbytes
           FROM sec,bt,stats
          WHERE sec.id       = bt.id
            AND sec.hid      = bt.hid
            AND sec.id       = stats.id
            AND sec.hid      = stats.hid
            AND sec.instance = stats.instance
");

[download]
This does only three physical file searches (one for each table) whereas your loop would do twice as many physical file searches as there are rows in the sec table (it would have to search each of the other tables for each row in sec). Please let me know if it improves your time and accomplishes what you want to accomplish

In reply to Re: How to make my DBD::CSV DB code faster. by jZed
in thread How to make my DBD::CSV DB code faster. by awohld

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.