Enforce Memory Cleanup

solegaonkar has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks I have a strange problem here. I read a huge file (1+ GB) 1000 records at a time. I read the records into hash map, process them and then try to free the memory using undef. Then process the next 1000 records and so on. But, strangely the code runs out of memory after some time. Looks like the garbage collection is not triggered in time. Is there some way I can enforce garbage collection? Have you faced such a problem? The code is something like this...


my %map

while (sysread(CSV, $record, 66)) {
        $map{substr($record, 18, 14)}->{substr($record, 3, 15)} = subs
+tr($record, 36, 29);
        if ($count++ > 1000) {
                &process();
                undef %map;
        }
}
[download]

Update:: The problem was solved by using scope rather than delete/undef... Updated the code to the following and that performs much better!!


while (sysread(CSV, $record, 66)) {
    my %map
    my $count=0;
    $map{substr($record, 18, 14)}->{substr($record, 3, 15)} = substr($
+record, 36, 29);
    while (sysread(CSV, $record, 66)) {
        $map{substr($record, 18, 14)}->{substr($record, 3, 15)} = subs
+tr($record, 36, 29);
        if ($count++ > 1000) {
                &process(\%map);
                last;
        }
    }
    &process
}
[download]

Thank you all for your help!!

Comment on Enforce Memory Cleanup Select or Download Code

Replies are listed 'Best First'.
Re: Enforce Memory Cleanup by Anonymous Monk on Jun 17, 2014 at 06:37 UTC
Looks like the garbage collection is not triggered in time. Not really :) Is there some way I can enforce garbage collection? Not really Have you faced such a problem? Yes, I have created circular references that leaked memory, once I fixed my mistakes the problem went away Read all about it, I did, and it improved my code tremendously :) Tutorials: Variable Scoping in Perl: the basics, Coping with Scoping , Mini-Tutorial: Perl's Memory Management, Lexical scoping like a fox, Read this if you want to cut your development time in half!, Closure on Closures , perlref#Circular References, Memory leaks and circular references, Circular references and Garbage collection., make perl release memory, about memory management , Re^3: Scope of lexical variables in the main script , Re: What's the weaken here for?, Devel::Cycle, Devel::NYTProf, Devel::Leak Devel::LeakTrace, WeakRef	[reply]
Re^2: Enforce Memory Cleanup by solegaonkar (Beadle) on Jun 18, 2014 at 03:33 UTC
Thanks for your reply, but I don't see any possible memory leakage here. I have attached the code above.. This one blows up with an empty process method ...	[reply]
Re^3: Enforce Memory Cleanup by Anonymous Monk on Jun 18, 2014 at 03:54 UTC
Thanks for your reply, but I don't see any possible memory leakage here. I have attached the code above.. This one blows up with an empty process method ... Well, in that code, %tmap keeps growing ; %tmap keeps eating memory ; until %tmap has eaten all the memory; does that make sense to you? ... $tmap ... &process(); ... That looks suspiciously like you're not coping with scoping, subs should take arguments ...	[reply]
Re^4: Enforce Memory Cleanup by solegaonkar (Beadle) on Jun 18, 2014 at 04:30 UTC
Re^5: Enforce Memory Cleanup by davido (Cardinal) on Jun 18, 2014 at 05:27 UTC
Re: Enforce Memory Cleanup by DrHyde (Prior) on Jun 17, 2014 at 11:42 UTC
I suggest that you reduce your code to the minimum possible that demonstrates the problem. In doing so you might solve the problem yourself, but if not then you'll have something useful to show us and then we can help you. You will also, of course, need to show us some sample data as well.	[reply]
Re^2: Enforce Memory Cleanup by solegaonkar (Beadle) on Jun 18, 2014 at 03:34 UTC
Thanks for your reply. I have attached the code above.. This one blows up with an empty process method. Can you see any possible problem here? Do I need to take special care because it is a hash of hash?	[reply]
Re: Enforce Memory Cleanup by Theodore (Hermit) on Jun 17, 2014 at 15:16 UTC
undefining a hash key will not remove its key, deleteing will do that.	[reply]
Re^2: Enforce Memory Cleanup by solegaonkar (Beadle) on Jun 18, 2014 at 03:37 UTC
Hi delete does clear the memory... but it takes ages to complete - since the code has to look through each key in the hash... We need some faster way to clear the memory	[reply]
Re: Enforce Memory Cleanup by Laurent_R (Canon) on Jun 17, 2014 at 18:01 UTC
Are you undefining the hash or the hash values? Deleting the keys would probably work, but I suspect that a proper use of scope would probably be faster at freeing the memory used by the hash. You should really show us your code for a better comprehension of what is happening and what to do.	[reply]
Re^2: Enforce Memory Cleanup by solegaonkar (Beadle) on Jun 18, 2014 at 03:45 UTC
Guess you are right.. Am I messing up things because of the scope here? Not very sure. I have added the code above. Can you please have a look at that - to see if there is something I am missing there?	[reply]