While the files seem pretty big if you look at number of lines, you can gain some comfort in knowing that even at 80 characters per line, and 50% overhead, that ten thousand line file will still only consume 1.2MB of RAM even if you slurped the whole thing into memory. Add to that the 6,000 line file with the same 80 characters and 50% overhead (another 720k), and you're consuming a whopping 1.92MB RAM if you slurp. If the ultimate in quick performance is necessary, and it's not possible to change the datasources to a database, slurp, but do so realizing that you'll run out of memory if those files get too big eventually.

If, on the other hand, if the files may end up growing substantially larger, Tie::File is a good way to go. The POD for Tie::File does state that slurping isn't going on. (The actual text says, "The file is not loaded into memory, so this will work even for gigantic files."

Another possible solution involves changing the design of the data sources. If the two files could be converted to a couple of database tables, scalability will not be a concern anymore.


Dave

In reply to Re: Suggestions re parsing one large file with elements of another large file by davido
in thread Suggestions re parsing one large file with elements of another large file by billie_t

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.