It should be noted that at no point does the code sort 14M arrays. The variable @rows actually holds the fields for the current line. The code sorts, for every column, the numbers of occurrences of different values. While a column could hold 14M different values in 14M lines, this is not the case here: With 14M lines of 1400 fields each, and 62GB in total, the average column has a data width of two bytes. You can only cram so much different values into two bytes (especially if it's text) - that's several orders of magnitude away from 14M and should fit into memory quite easily.

The sorting problem in the code is, as has been pointed out, that the sorting is done 14M times instead of once.

In reply to Re^2: efficient perl code to count, rank by haj
in thread efficient perl code to count, rank by Perl_Noob2021

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.