You haven't told us what this file is and what you are doing with it. I'm guessing that you are doing repeated searches through this huge thing. It sounds like you don't have enough RAM on your system. It could be that the easiest answer for you is to spend $50 on more RAM!

For example, I have an app that searches a data set of 700 files. The first exhaustive search thru all 700 files takes about 7-8 seconds. The second time I search thru those same 700 files, it takes way less than 1 second! Wow! My system has plenty enough ram for the data in all those files and my OS can use a whole bunch of the RAM for its own use. What happens is that on the first search, the OS winds up caching all of the files in memory. The second search is 10x+ faster because the OS knows that those files haven't changed and it has a cached copy of them.

In my app the searches are driven by interactive user input so I can't bunch them all up and do them at the same time. Of course there all sorts of fancy database things that I could do. But there is no need! Typical user works with this program for a couple of hours at a time. Actually nobody has even noticed that the first search is slow! They just remember how well its been working for the last 30 minutes! Anyway I can deliver sub-second response time to a user request and that is way fast enough! It takes the user way longer than that to figure out what the next question is gonna be! No need to optimize further! I would add that this app gives a "progress report" as it searches and delivers partial results as they become available and the user can abort the operation if needed. Even if search n+1 took a few seconds, this is ok as user is starting to think about the results that are flowing in.

Anyway I have very simple code that just gets faster the more times it is used. Perfect. Don't do something yourself that the OS can also do very, very well and with no coding effort on your part!

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.