Boetsie has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,

i have a huge hash which is made in a function and deleted in a function. The function reads a file, checks if the line was already present, if not it prints to a new file and otherwise it is not printed to a file. It doesn't return anything. The whole process takes about 30% of memory. If the function is ended and i check my memory, it's still 30% memory. When i run another function, also requiring alot of memory, i want this memory to be removed first. I now do it with a pause of 30 seconds, but this is a weird solution. Someone has an idea how to do this?

Thanks in advance.

Replies are listed 'Best First'.
Re: wait till memory is flushed
by BrowserUk (Patriarch) on May 19, 2010 at 11:08 UTC
    If the function is ended and i check my memory, it's still 30% memory. When i run another function, also requiring alot of memory, i want this memory to be removed first.

    The thing is, requesting memory from the system is a costly operation. Far more costly than re-using memory you all ready have. So, if after the first function runs you have sufficient memory free for use within the process for the second function to run, it will run more quickly than if you freed that free memory back to the OS and then have to go cap in hand and ask for it back.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: wait till memory is flushed
by moritz (Cardinal) on May 19, 2010 at 10:32 UTC
    Perl doesn't always give the memory back to the operating system. (In fact I don't remember the conditions for when it does).

    But you can still use it inside the same perl process for something else that needs much memory.

    So my advise for you would be not to care about the memory for now, because perl is supposed to handle it for you.

      If i pause for 30 seconds the memory is empty (<1%) again. So the 30% memory will probably remove after some time? The reason for the question is that if we have even more data, we are worried that the second function canīt run because of too few memory left.

        I'm sorry if I haven't been clear before; I'll try to explain it with more words this time.

        Let's say you have 1G of memory; you build a huge data structure in a hash; for that perl allocates 300M of memory from the operating system.

        You store that hash in a lexical variable my %h = .... When this variable goes out of scope, perl internally marks the memory as "not needed anymore". However it doesn't give the memory back to the OS, because it suspects that you might use more memory later on in your program.

        Now if you wait for 30s, perl decides that it can give the memory back, because apparently you don't need it anymore.

        On the other hand if you don't wait, but allocate another huge data structure, perl will use the memory that %h occupied before, and only when that is exhausted it will ask the operating system for more memory.

        So if your program has the form

        * use up 300M of RAM * let the variables go out of scope that point to the memory * build a data structure that needs 400M

        then the last step only increases the memory usage of the perl script by 100M, because it uses the 300M first that was marked as not-needed-anymore.

        This is why I wrote that you shouldn't bother to sleep before doing the second task - perl will not waste your memory, unless you do something stupid (like keep a reference to the first, huge hash).

        I tried this code. It allocated 13% of my memory at the beginning, and it stayed at this level until the end, which means new invocation of the function does not increase the memory consumption.
        sub test { my $size = 1000000; my %hash; @hash{1..$size} = (rand()) x $size; sleep 5; print keys %hash; } test(),sleep 5 for 1..10;
Re: wait till memory is flushed
by cdarke (Prior) on May 19, 2010 at 11:06 UTC
    I am curious, how do you "check my memory"?

    I want this memory to be removed first Why? Perl, like most systems, will reuse allocated memory if it can - that is more efficient than freeing it and then allocating again. Very few systems allow an application to return memory to the OS anyway. It is, after all, probably virtual memory (although you don't say which operating system you are using), so it will not be locked in RAM.

    If you are really worried about this then run the hash processing in another process, the memory will be freed and returned to the OS when that process ends. You can save the hash using something like Storable.
Re: wait till memory is flushed
by Fletch (Bishop) on May 19, 2010 at 13:06 UTC

    Another possibility would be to use something like BerkeleyDB (or GDBM_File or whatnot) and leave the hash on disk instead of in memory.

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

Re: wait till memory is flushed
by choroba (Cardinal) on May 19, 2010 at 10:24 UTC
    Do you use my when declaring your hash inside the sub?
      yes i do. is that ok?
Re: wait till memory is flushed
by f77coder (Beadle) on Sep 01, 2014 at 06:18 UTC

    Old thread but I was looking for perl memory flush as well. On OSX, I found an app that monitors memory and when perl starts to hog too much, I force it to flush. memory clean from fiplab