I definitely agree with this. Most of the files I am working with would give me the "out of memory" message if I tried to slurp then entirely into memory. They are just way too large. Processing one line at a time is quite often the solution. But I can't say more without seeing the program and a data sample.

Naveen_learn:

Perhaps it's time to add more memory to your machine. If you've already maxed out your memory, then maybe it's time to get a machine that can use more memory and use a 64 bit OS so you can use all the RAM you add to your machine.

...or you could change your program to use less RAM. But since I can't see your program, you'll have to figure that out yourself.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

I can't believe how many responses say to just add more memory. I realize this might be tongue-in-cheek, but the responsible thing for a developer to do is to figure out how to consume less memory.

We haven't seen any code yet, nor any sample input. Thus, the question is mostly unanswerable. But attempts at providing suggestions for an unanswerable question ought to at least lean in the direction of what a developer might do to solve the problem, rather than what a hardware tech might do.

If at all possible, read your input one record at a time, process that record, write out anything that needs to be written, accumulate whatever needs to be accumulated, and then discard from memory the current record and move on to the next. Some variation on this theme is how most of these scalability issues are resolved. Maybe you need to hold onto two or three records. Or maybe you can immediately drop any records that are uninteresting and only hold onto ones that need further work. But in the end your approach will probably look something like this.

Another technique might be to break up the list of records into smaller chunks, and work on them individually. Accumulate results for each chunk, and in the end, compile results for all the accumulated data. This is a "map-reduce" strategy.

So these are some very generalized suggestions, which are about as specific as possible given the ambiguities of the original question.

To the original poster: Please consider this: When you present a question that is devoid of detail, it usually means that you have become so exacerbated with the problem that you've given up trying to figure it out yourself. But we have no chance of figuring it out either unless presented with those very details that have so frustrated you.

I agree with your file processing strategies(++), but wanted to add an alternative:

If you need to hold a number of inter-related records in memory, you should consider using a database - since they are designed to optimize memory usage, and provide rapid access.

             My goal ... to kill off the slow brain cells that are holding me back from synergizing my knowledge of vertically integrated mobile platforms in local cloud-based content management system datafication.

FYI - this is also known as configuring your "swap" file.

Chips might be cheap (and the cost of labor substantially lower) but it has been my experience that such a stopgap rarely avoids the need to bite the bullet and do a proper design/redesign in the end.