Chief of Chaos has asked for the wisdom of the Perl Monks concerning the following question:

Fellow monks,

I wrote a big two component application (client-server) which detects some database table differences.
The client holds 2 hashes with the query results.
With these hashes it creates files for the user with the missing and the different rows.

But what happens when it writes the data in the hashes and there is not enougth memory available?
In my experimantal study the process works with a whole cpu and did no further logging.
So I thing it tries (as fast as possible) to allocate memory.

Is it possible to get the size of the memory of a hash/array/variable at runtime, so I can intervene somehow?

In some docu I read that the memory of a process can not be freed while the process is not terminating. So there is no chance to free a hash/array/variable of one application to get more mem for another application. Is this correct?

Best regards,

Replies are listed 'Best First'.
Re: Memory Allocation
by broquaint (Abbot) on Feb 17, 2003 at 11:10 UTC
    Is it possible to get the size of the memory of a hash/array/variable at runtime, so I can intervene somehow?
    If you want to find you how much memory your datastructures are using at run-time check out Elian's Devel::Size.
    the memory of a process can not be freed while the process is not terminating
    For most (if not all?) modern OSs this is the case. Until a process exits its allocated memory will not be freed.


      Devel::Size is far from accurate when it comes to complex hashes, beware.

      My recent project which had some 140,000 hashes inside another hash reported its size significantly differently between Linuxes 'top' and Devel::Size.
      While I am not entirely sure why this is the case, it still happened; top would say the programme was at 70MB resident and Devel::Size would say 40MB, and I can assure you the actual code itself manipulating the data did not occupy all of 30MB.

      At any rate, your mileage may vary - I just wouldn't use Devel::Size to be particularly accurate without evaluating exactly what type of data structure you are using, and comparing what the OS thinks the resident size is, and what Devel::Size thinks it is.

      -- Alexander Widdlemouse undid his bellybutton and his bum dropped off --

        While it's possible I've missed something, and I'll go check, Devel::Size is generally an accurate representation of the amount of actual memory that perl uses for its internal data structures. This does not mean that it will give you an answer anywhere near what top will give you--far from it in some cases.

        It can't, for example, show you the memory that might've been allocated by XS code and not managed by perl. (The expat XML library does this, as do the various DBD modules, for example) At some point I may put in support for a few known modules or libraries (like expat, which some of the XML modules use) but that's a dodgy thing, as it means Devel::Size needs grubby details of other people's internals as well as perl's.

        It also doesn't show you memory your process (but not perl) may have allocated but that isn't currently tied up in a perl data structure. Memory allocations are generally done with a particular granularity, often a power of two or a factor of four or eight (depends on the type of request, OS, and processor architecture), and if your perl has requested a chunk of memory smaller than that amount it'll get silently rounded up, but the extra won't be usable, though it is put back on the free list if the used bit is freed.

        The C RTL also keeps track of allocated and freed memory, since freed memory is rarely actually given back to the system. While the RTL tries to satisfy requests for new memory from the pool of freed chunks, this isn't always possible, and it's not too tough to accumulate a lot of little bits of memory that are too small to be of any use. The more alloc/free cycles you have, the more likely you are to have a significant fragmentation of your memory pool, which usually means lots of allocated but unusable memory.

        A single big alloc with a resulting free can also blow your process' memory usage way out, even if you're no longer actually using the memory--it's on your process' free list, ready to be allocated and still counting against your memory size, but Devel::Size won't be able to find it.

        And, of course, I may well have blown it somewhere. Wouldn't be the first time and, like I said, I'll go check and see. I'll also go update the readme for it so folks know what Devel::Size's numbers are good for, and what they aren't. A general set of docs on memory allocation is probably also in order.