I don't understand the question.

The first snippet doesn't do anything useful. It unpacks things, then throws away the results. You might as well not open the file at all. A program like that will run in approximately zero seconds. Much faster!

As for the second snippet, yes, there's memory allocation for each iteration. That's because unpack creates and you assign new values to @data for each row. You can't get around that if you want to do something with the data. And, depending on your data structure, unpack is probably the fastest way to get at it.

Assuming your code handles exactly 2.1 million rows and the code runs in 70 seconds, that's 30,000 iterations per second. That's pretty fast.

This falls in the category of "things you can't optimize away without breaking the program" -- you're not using regular expressions to get at the data, which would slow you down, and you're not using split, which is probably slower than unpack in this case.

You're probably as fast as you can get without removing anything useful.

In reply to Re: Assining data to an array is slow.. by chromatic
in thread Assining data to an array is slow.. by smferris

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.