Re^2: perl using more Memory Than I have Got

Er, what's vague about it? :-) Am I missing something?

Obviously I'd like to use less memory, but I cant in the short term. Is the group wisdom to NEVER let your perl program use more memory than you have physical memory? Even so - I'd like to understand what is happening to the process right now. > What kind of processing are you doing?

Well, I am processing the results of several very large database queries. These need to go in various very large hashes. I've been looking at out of memory caches but the network IO will slow things down even more.

I would like to break up the task into smaller chunks if I could but there doesnt seem to be an easy way of doing that.

Comment on Re^2: perl using more Memory Than I have Got

Replies are listed 'Best First'.
Re^3: perl using more Memory Than I have Got by stiller (Friar) on Feb 28, 2008 at 18:14 UTC
Hi again, the vague part is details about your proccessing that could help suggest alternative aproaches. I do not represent the group wisdom in these parts, I'm representing nobody but me. I'm in the lucky position that I've seldom run out of memory, but I have had to consider ways to reduce memory footprint. Once your machine start swapping, you have hit a very real speed bump. How bad the degradation is, depends on factors like your operating systems swap strategy, swap disk speed etc. Also it matters a lot if your process keeps refering back to already swapped out data or not. Another set of RAM chips might be an inexpensive solution compared to the time needed to develop a better software solution.	[reply]
Re^3: perl using more Memory Than I have Got by alexm (Chaplain) on Feb 28, 2008 at 21:37 UTC
Well, I am processing the results of several very large database queries. These need to go in various very large hashes. I've been looking at out of memory caches but the network IO will slow things down even more. I don't know for sure how DBM-Deep would perform in this case, but it's been mentioned several times in other nodes regarding large hashes. I'd give it a try.	[reply]