Ok here is the background: I'm slurping in data from a file which contains protein sequences in FASTA format. I am then looping through and pulling out each protein header and its sequence and then generating some n-gram data and storing it into a BerkeleyDB file. In case you are wondering I am using an XS-based module called Text::Ngram to do the n-gram generation. It returns a reference to a hash containing the n-grams and their frequencies. Here is a code snippet... 
use Text::Ngram qw( ngram_counts );
use ...
use ...

my $db = "/home/db/repository/megalith.db";
my $ngram_width = 5;
my %protein_search_hash;

unlink $db;
tie %protein_search_hash, 'MLDBM', -Filename => $db, 
                                   -Flags    => DB_CREATE | DB_INIT_LO
+CK
         or die "Cannot open database '$db: $!\n";

open FILE, "<test_data_2MB.txt" 
    or die "Failed to load test_data: $!\n";
my $fasta_sequence = do { local $/; <FILE> };
close(FILE);

# Precompile regex for speed
my $regex = qr/>([\S]*)\s*.*\s([A-Za-z\s]+)/;
my $regex_d = qr/^>[\S]*\s*.*\s[A-Za-z\s]+/;

# PARSE $fasta_sequence and split header from the sequence
while( $fasta_sequence =~ m/$regex/igm ) {
    my $sequence = $2;
    my $header = $1;
    $sequence =~ tr/[\r|\n]//d; # delete all line breaks
    # generate n-grams
    my $ngram_href= ngram_counts($sequence, $ngram_width);
    $protein_search_hash{$header} = $ngram_href;
}
...
...
...

[download]
My problem is with memory usage: it continually gobbles RAM. Even if I comment out 
    $protein_search_hash{$header} = $ngram_href;

[download]
...it still eats the same amount of RAM each time. However, I have narrowed it down to 
    my $ngram_href= ngram_counts($sequence, $ngram_width);

[download]
...as being the problem. I've tried undef and putting it in a subroutine but nothing stops the insatiable thirst for RAM. Currently, I'm limited to processing data files under 13MB (anything higher and I'm out of RAM and into swap). If I comment out the above line, RAM usage flatlines like I would expect. Why isn't '$ngram_href' being overwritten each iteration through and the old value being GC'd? Monks, what am I missing here? Any help would be much appreciated!

In reply to Memory Growth Problem by Robgunn

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.