You haven't told us what this file is and what you are doing with it. I'm guessing that you are doing repeated searches through this huge thing. It sounds like you don't have enough RAM on your system. It could be that the easiest answer for you is to spend $50 on more RAM!

For example, I have an app that searches a data set of 700 files. The first exhaustive search thru all 700 files takes about 7-8 seconds. The second time I search thru those same 700 files, it takes way less than 1 second! Wow! My system has plenty enough ram for the data in all those files and my OS can use a whole bunch of the RAM for its own use. What happens is that on the first search, the OS winds up caching all of the files in memory. The second search is 10x+ faster because the OS knows that those files haven't changed and it has a cached copy of them.

In my app the searches are driven by interactive user input so I can't bunch them all up and do them at the same time. Of course there all sorts of fancy database things that I could do. But there is no need! Typical user works with this program for a couple of hours at a time. Actually nobody has even noticed that the first search is slow! They just remember how well its been working for the last 30 minutes! Anyway I can deliver sub-second response time to a user request and that is way fast enough! It takes the user way longer than that to figure out what the next question is gonna be! No need to optimize further! I would add that this app gives a "progress report" as it searches and delivers partial results as they become available and the user can abort the operation if needed. Even if search n+1 took a few seconds, this is ok as user is starting to think about the results that are flowing in.

Anyway I have very simple code that just gets faster the more times it is used. Perfect. Don't do something yourself that the OS can also do very, very well and with no coding effort on your part!

In reply to Re^4: Most efficient way to load file contents into scalar? by Marshall
in thread Most efficient way to load file contents into scalar? by smallwalrus

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.