Re: How to reslove this memory issue

This is a common misunderstanding of the output of free. The memory statistics you showed do not indicate excessive memory use. On the contrary, they show only about 2.3% of system memory in active use by programs.

If you look more closely at the first line of numbers, you will see that 7928420 of the 8124068 memory pages in use are being used to cache data. This memory is, for all intents and purposes, free. Your operating system it using it to improve system performance, but can throw the cached data out and reallocate the memory for other uses more-or-less instantly if the memory is needed elsewhere.

To see whether you actually have a memory shortage, you need to look at the second line of numbers, which adjusts the totals by treating buffers and cache as free memory rather than used (because, again, if needed, that memory can be reassigned instantly). Looking at that line, you will see that only 182400 pages are in use for "real" data, leaving 7975776 available.

Comment on Re: How to reslove this memory issue Download Code

Replies are listed 'Best First'.
Re^2: How to reslove this memory issue by locked_user sundialsvc4 (Abbot) on Sep 11, 2014 at 13:32 UTC
One possible thing to consider, though, is that if this command was executing on a server that was doing many other things at the same time, we don’t necessarily know who is using those buffers, nor what distribution of buffer use is needed, on the system as a whole, to make everything run efficiently. When the OP said that the activity had a serious impact, I believe him. Stealing a buffer is not a no-cost operation, because now it means that you must do another physical disk-read to re-obtain whatever it once held. Even though I’m just on my first cup of coffee this morning, I also have a healthy skepticism about these numbers. They seem much too small, given the reports of what this program is supposed to be doing and how much memory it ought to be taking up. You really need to profile this thing, to see what it’s waiting on ... how much disk I/O, how many page faults and so on. How much virtual-memory it wants vs. what its resident page-set size is, under actual load conditions. It could well be that a database-oriented solution (which can basically do a job like this with a `JOIN`) might be a better solution for two reasons. First, that a “memory oriented” solution might well still wind up doing an equivalent amount of disk I/O ... especially if several other big programs are running elsewhere ... and because, with two tables and a JOIN, this requirement would probably be completed by now. After all, of this we can be sure: soon, there will be four more fields to add; then, Marketing will ask for another six. So it goes. This approach seems to be leading quite rapidly to a corner-box, and is rapidly losing its “shine.”

Replies are listed 'Best First'.

Re^2: How to reslove this memory issue
by locked_user sundialsvc4 (Abbot) on Sep 11, 2014 at 13:32 UTC

One possible thing to consider, though, is that if this command was executing on a server that was doing many other things at the same time, we don’t necessarily know who is using those buffers, nor what distribution of buffer use is needed, on the system as a whole, to make everything run efficiently. When the OP said that the activity had a serious impact, I believe him. Stealing a buffer is not a no-cost operation, because now it means that you must do another physical disk-read to re-obtain whatever it once held.

Even though I’m just on my first cup of coffee this morning, I also have a healthy skepticism about these numbers. They seem much too small, given the reports of what this program is supposed to be doing and how much memory it ought to be taking up. You really need to profile this thing, to see what it’s waiting on ... how much disk I/O, how many page faults and so on. How much virtual-memory it wants vs. what its resident page-set size is, under actual load conditions.

It could well be that a database-oriented solution (which can basically do a job like this with a JOIN) might be a better solution for two reasons. First, that a “memory oriented” solution might well still wind up doing an equivalent amount of disk I/O ... especially if several other big programs are running elsewhere ... and because, with two tables and a JOIN, this requirement would probably be completed by now. After all, of this we can be sure: soon, there will be four more fields to add; then, Marketing will ask for another six. So it goes. This approach seems to be leading quite rapidly to a corner-box, and is rapidly losing its “shine.”