Some real low-hanging fruit that should roughly halve your memory requirement:

You accumulate all output in @output and only write it out at the end. Instead, eliminate the variable @output and instead of pushing there, write to the output file directly.

You keep the arrays @IDs, @lengths, @IDsLens and the values in them around for the whole runtime of your program. Insteadd, empty out those arrays as soon as you don't need them anymore. This is usually easiest done by restricting the scope of such variables to be the minimal necessary scope. For example the @IDsLens array is not needed after the assignemtn to the hash.

For futher improvement, instead of building up @IDsLens and then assigning it to the hash, eliminate the variable @IDsLens completely and do the hash assignment in place:

    #push @IDsLens, $ID2;
    #push @IDsLens, $len2;
    $IDLen_hash{ $ID2 } = length( $ID2 );
...
    #my %IDLen_hash = @IDsLens;

[download]

You should test whether storing all the lengths in a hash is worth the improvement, compared to directly calculating the length of the string each time, because you only use that hash once to retrieve the length again.

In reply to Re: Reduce RAM required by Corion
in thread Reduce RAM required by onlyIDleft

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.