Simple rule: when you have data, think database. I don't care if it's mysql, postgres, Db2, sqlite, but use a db.

In this particular case, especially, you'll probably find it runs faster to load the data, ask the db to give you the output, and discard the db than it is to try to do it in memory. Databases are often optimised for this type of data access, and will be a much more efficient (time-wise especially) use of disk than swapping / paging. Sqlite probably will suffice, but, if not, mysql and postgres are free, and not difficult to install on most linux distros, and not really much harder on Windows, even without WSL.

I know, you asked a perl question, and there's not a lick of perl in this answer. But the reality is that while you can do this in any turing-complete language, you still should use the right tool for the job, and, in this case, a database is that right tool.

In reply to Re: efficient perl code to count, rank by Tanktalus
in thread efficient perl code to count, rank by Perl_Noob2021

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.