cache some data for speed reasons, but after a while it starts to consume huge amounts of memory.Sounds like your caching algorithm needs improvement. From watching long lived Perl programs, a typical pattern is that memory usage increases until some maximum level is reached and then it stays there. Perl does not ever release memory back to the OS. Once Perl has it, it keeps it. However, Perl can re-cycle memory for its own use when you tell Perl that some memory is no longer in use.
Perl uses reference counting to decide whether some piece of memory is still in use or not. If you have a reference to some array, undefine that reference to tell Perl that you aren't using it anymore, Perl will re-use that memory provided that there isn't some other reference to that memory.
If your cache remembers everything for all time, then compressing the data just delays the inevitable. Eventually you will run out of memory if you keep generating stuff that is cached without removing anything from the cache!
If this data is "stable", meaning usable for a very long time, then some DB solution might be appropriate. Another thought, I guess a simple minded approach would be to wait until the cache reaches a certain number of things in it and then "clear", undef the values in the cache. This won't release the memory back to the OS, and Perl will re-use it. So your program would "hick-up" and get slower for awhile when that happened as each value would have to be re-computed. But along with zapping everything, data that hadn't been used for a very long time would get "cleaned out" also.
More advanced schemes would have to keep track of when some data was last used. I'm not sure in your app how difficult that would be to implement. Another thought, I don't know how well these various "Tie" modules work, but the idea is that an array is mapped to some disk file and some subset of the data is kept in memory. If things work well, then only the "recent" stuff in this file will wind up being memory resident. I haven't experimented with this and I don't know how well it works, in particular how to regulate the max memory that the "tied" file uses in memory. But in this type of scheme, the file would continue to get larger, but memory footprint would not.
It would be helpful if you could explain more about your application. If we can compress the data by 1/2, then that only "puts off the day of reckoning" by a factor of 2, so I suspect that is not a great solution.
This statement: "after a while it starts to consume huge amounts of memory" appears to be key. What do you know about why that could be happening? This seems to mean that it works "ok" for awhile and then something "new" happens. What do you as the app developer think that is? What kind of problem symptom happens? Consuming a "huge amount" of virtual memory may or may not be a problem. The OS may be "saving your butt" by swapping parts of memory out to disk.
If your problem is: "I have a process that continually grows memory requirements without bound", no amount of "compressing the data" is going to solve that problem, just delay it.
In reply to Re: array overhead
by Marshall
in thread array overhead
by Anonymous Monk
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |