perl using more Memory Than I have Got

alexmc has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: perl using more Memory Than I have Got by Joost (Canon) on Feb 28, 2008 at 19:22 UTC
Anyone here use processes which take up more memory than you have on the box? Sometimes. If I know that might happen I set the max memory limit for the program at something sane (i.e. less than about 75% of the total amount of RAM). That will kill the process if it tries to get more memory, but since a process that takes up more RAM than you actually have will probably bring the whole system grinding to a halt, it's usually the correct response. Then I try to fix the code so that it doesn't take as much RAM. update: or add another couple of GB of RAM. I'm currently working on a system that takes about 4 to 8 GB of RAM. Works perfectly fine on a 16 Gb machine. It doesnt seem to be swapping much. The program is still running - but incredibly slowly. Trust me, the swapping is the cause of the slowness. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]
Re^2: perl using more Memory Than I have Got by alexmc (Novice) on Feb 29, 2008 at 08:50 UTC
Big thanks to everyone who responded.	[reply]
Re: perl using more Memory Than I have Got by locked_user sundialsvc4 (Abbot) on Feb 28, 2008 at 18:37 UTC
Your program is “running incredibly slowly” because it is thrashing horribly ... and it's never going to get better. You're either going to have to re-design your algorithm or acquire more physical RAM for the box. In operating-system circles we speak of a process's working set, this being the (constantly changing) set of 4K pages which the process needs to have in-memory in order to run without delay. The working-set size of any process needs to be a manageable fraction of the total virtual-memory space, and the working-set needs to be fairly non-volatile, that is to say, once the initial pages have been paged-in the amount of page-replacement activity needs to drop. A program with a 10GB “footprint” is either going to have to be re-designed, or run on a machine that has 15GB or 20GB of physical RAM. Yes, this might be a case where the right thing to do is to “throw silicon at it,” and if that be the case, make it so.
Re^2: perl using more Memory Than I have Got by alexmc (Novice) on Feb 29, 2008 at 08:48 UTC
> it is thrashing horribly Well that is what I would expect - except it isnt. I dont know why.	[reply]
Re: perl using more Memory Than I have Got by stiller (Friar) on Feb 28, 2008 at 17:36 UTC
It's hard to give solid advice based on the vague information you provide. But one sensible thing you can do is to take control over what is put to disk and when, don't leave that to the OS. That is, if you can't slim down your datastructures, write to disk in stead of memory. What kind of processing are you doing?	[reply]
Re^2: perl using more Memory Than I have Got by alexmc (Novice) on Feb 28, 2008 at 17:57 UTC
Er, what's vague about it? :-) Am I missing something? Obviously I'd like to use less memory, but I cant in the short term. Is the group wisdom to NEVER let your perl program use more memory than you have physical memory? Even so - I'd like to understand what is happening to the process right now. > What kind of processing are you doing? Well, I am processing the results of several very large database queries. These need to go in various very large hashes. I've been looking at out of memory caches but the network IO will slow things down even more. I would like to break up the task into smaller chunks if I could but there doesnt seem to be an easy way of doing that.	[reply]
Re^3: perl using more Memory Than I have Got by stiller (Friar) on Feb 28, 2008 at 18:14 UTC
Hi again, the vague part is details about your proccessing that could help suggest alternative aproaches. I do not represent the group wisdom in these parts, I'm representing nobody but me. I'm in the lucky position that I've seldom run out of memory, but I have had to consider ways to reduce memory footprint. Once your machine start swapping, you have hit a very real speed bump. How bad the degradation is, depends on factors like your operating systems swap strategy, swap disk speed etc. Also it matters a lot if your process keeps refering back to already swapped out data or not. Another set of RAM chips might be an inexpensive solution compared to the time needed to develop a better software solution.	[reply]
Re^3: perl using more Memory Than I have Got by alexm (Chaplain) on Feb 28, 2008 at 21:37 UTC
Well, I am processing the results of several very large database queries. These need to go in various very large hashes. I've been looking at out of memory caches but the network IO will slow things down even more. I don't know for sure how DBM-Deep would perform in this case, but it's been mentioned several times in other nodes regarding large hashes. I'd give it a try.	[reply]
Re: perl using more Memory Than I have Got by almut (Canon) on Feb 28, 2008 at 19:04 UTC
Should I learn about any other tools? You haven't mentioned which tools you know already, so maybe I'm not telling you anything new :) Anyhow, `vmstat` would certainly be useful for figuring out if the box is paging (option `-p` will give details on paging activity). `iostat` might also be useful (for checking disk I/O, etc.) To get an idea of what the progam is actually doing (systemcall-wise), I'd use `truss` first. Dtrace is definitely nice, too, but it might take quite a bit of reading to get going... Generally, I'd second what stiller said: get more physical RAM, if possible. If your program does need more memory than available (and this is something you can't change), it might well turn out that no amount of system analysing will actually help to make things faster (at least not significantly). Sure, you may have learned a lot in the end, but it's not too unlikely that the program will still be slow... :)	[reply] [d/l] [select]