How often should I flush?

elusion has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monks,

In my quest to more fully understand math and perl, I've created a program that goes through an infinite loop of numbers (that gets bigger each time) and it does some stuff to it using prime numbers. I'm hoping to speed this up using the module memoize. It's a lot faster, I've already tested it. It makes some calls to the sub prime each time through the loop, so I memoized it. Right now the number's in the 9_000_000's. It goes through about 50 numbers a second.

The question is: How long will it take before my computer's memory is full? or How often should I flush the cache? The last thing I want is for my computer to crash from memory. Thanks for the help,

- p u n k k i d
"Reality is merely an illusion, albeit a very persistent one." -Albert Einstein

Comment on How often should I flush?

Replies are listed 'Best First'.
Re (tilly) 1: How often should I flush? by tilly (Archbishop) on Feb 20, 2001 at 08:26 UTC
Two options. First of all in Memoize you can choose to have the cache be tied to a dbm on disk. That will allow your cache to grow on disk, which you probably have more of than RAM. The other option to consider is to use the plug-in Memoize::Expire which allows you to set various kinds of expiration policies for your cache. Play around with that. I would suggest an expiration policy of once every X accesses where X is relatively large (eg a million or so). YMMV.	[reply]
Re: How often should I flush? by AgentM (Curate) on Feb 20, 2001 at 05:50 UTC
Flush when you're done. No seriously, I mean it. If you run out of memory, Perl will be sure to kick the bucket and say "Out of memory!" You shouldn't worry about it too much- it doesn't hurt anything. If you find that after some X iterations, your script runs out of memory, take note of it, insert a courtesy flush at this point, and strike a match. AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.	[reply]
Re: Re: How often should I flush? by elusion (Curate) on Feb 20, 2001 at 06:04 UTC
The point is that I don't want that to happen. I want it to be running 24/7. I'm only home for a few hours, so if it stops sometime else I lose hours of data. I want to prevent that. Is there a way to detect when Perl's just about out of memory? - p u n k k i d "Reality is merely an illusion, albeit a very persistent one." -Albert Einstein	[reply]
(jptxs): Re: How often should I flush? by jptxs (Curate) on Feb 20, 2001 at 08:12 UTC
If you really are concerned, write a script and use vmstat(see the man page, but Always ignore the first line of output from vmstat) to watch the memory use of the program and have it issue some "flush" command to your program if it reaches a certain threshold. That way you can enjoy not having to make the program have to worry about it, but also not need to worry about missing the event if it happens. `"A man's maturity -- consists in having found again the seriousness one had as a child, at play." --Nietzsche` [download]	[reply] [d/l]
Re: How often should I flush? by mothra (Hermit) on Feb 20, 2001 at 19:26 UTC
Dominus's excellent article Suffering from Buffering may give you not only the answer you're looking for, but even an understanding of why it works. :)	[reply]
(ichimunki) odd link by ichimunki (Priest) on Feb 20, 2001 at 20:54 UTC
If you read both the question and the article, they have little in common. The question is about memory, the article is about I/O.	[reply]

AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.