Dear Monks

I wish to be enlightened by your collective wisdom once more, please

I've tried to simplify the problematic area of my code below

The trouble I am experiencing is that between the loops, even though I am resetting my hashref, I don't seem to be freeing up the memory it occupied.

If you run the code while looking at "top" in Linux or some equivalent, you'll see that after the first loop, even though the hashref is reset on line 89, there's still a fair chunk in memory

I've no idea what I'm doing wrong here. PLEASE can someone tell me what I'm doing wrong.

I've commented the code to show you what it is doing and note that this code will take about 25seconds (cos of the sleeps) and use up about 4GB RAM. If you need it to use less of both then adjust the sleeps and decrease the values on lines 12 and 13

MANY thanks in advance for any advice
Kind regards,
Rich

#!/usr/bin/perl

use strict;
use Devel::Size qw(total_size);

# creating a reference to an anonymous hash
my $coverage = {};
# I'm going to run a loop twice with these values
my @times = qw(once twice);
# some data to use to fill an array in the sub
my %lengths = (
 "id1" => 50000000,
 "id2" => 50000000,
 );

# keeping track of the estimated memory size of the coverage hashref
my $total_size = total_size($coverage);
print "size before the loops = $total_size\n";

# OK run the sub twice with different parameters
foreach my $t (@times){

 if ($t eq "once"){
  # I'm doing stuff here that's conditional on $t
  # then running the sub
  &doWork($t);
  }
 elsif ($t eq "twice"){
  # I'm doing stuff here that's conditional on $t
  # then running the sub
  &doWork($t);
  }

 # keeping track of the estimated memory size of the coverage hashref
 my $total_size = total_size($coverage);
 print "size after $t sub = $total_size\n";

 # reset the hashref so that memory should be small again
 %{ $coverage } = ();

 # keeping track of the estimated memory size of the coverage hashref
 my $total_size = total_size($coverage);
 print "size after resetting the hashref = $total_size\n";
 # while total_size reports that the hashref is small, top shows that 
+the program is still hogging memory
 # what am I doing wrong????
 print "what's the mem doing?\n";
 sleep(10);
 }
sub doWork {
my $time = shift;

# keeping track of the estimated memory size of the coverage hashref
my $total_size = total_size($coverage);
print "size at the start of $time = $total_size\n";

# the main work in the sub is to fill a large reference to an array
foreach my $l (keys %lengths){
 print "creating $l\n";
 push @{ $coverage->{$l} }, 0 for(1 .. $lengths{$l});
 }

# keeping track of the estimated memory size of the coverage hashref
my $total_size = total_size($coverage);
print "size after cov made in $time = $total_size\n";
sleep(5);

# then the sub goes off and does some work that gives back some coordi
+nates to use to increment particular values in the array created abov
+e
# get some data to work with
my %increments = {
 "id1" => qw(400-500,6000-6500,100000-100500),
 "id2" => qw(400-500,6000-6500,100000-100500)
 };

# then the values in the array are incremented and passed back to the 
+main program to do something with it
print "incrementing\n";
foreach my $i (keys %increments){
 foreach my $p (@{ $increments{$i} }){
  my ($s,$e) = split("-",$p);
  $coverage->{$i}->[$_]++ for ($s - 1 .. $e - 1);
  }
 }

# keeping track of the estimated memory size of the coverage hashref
my $total_size = total_size($coverage);
print "size after increments made in $time = $total_size\n";
sleep(5);
}

[download]

In reply to Unexplained memory hogging by richardwfrancis

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.