I had a data file that might as well have been generated this way: 

for( my $i = 0; $i < 1_000_000; $i++ ) {
    print join ' ', map { int rand 256 } 0 .. 50;
    print "\n";
}

[download]

I read my data file this way: 

my @d; $#d = 1_000_000; # presize the array!
my $i = 0;
while (<>) {
    chomp;
    $d[$i++] = [ split ];
}

[download]

After that, I do a lot of arithmetic with the resulting array. I expected this to take a lot of memory and a long time, but I did not expect my memory usage to increase once the array was populated. Much to my dismay, my program kept growing and growing until I killed it to save its slimmer siblings.

I spent a good hour commenting out parts and printing out pieces before I got around to blaming the arithmetic, and it finally came clear to me. My huge pile of strings (hot off the text file!) was being converted to numbers.

So now my read loop has this in it: 

    $d[$i++] = [ map { int $_ } split ];

[download]

As a result, my program uses a lot less memory, and, more importantly, it does not grow.

In reply to Strings and numbers: losing memory and mind. by kyle

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.