Unfortunately Perl lacks a good memory profiling tool. I've considered working on one but it's a challenging task - Perl's memory allocation system is a complex beast.

-sam

I am relatively certain that the leak occurs somewhere within these lines of code, however the code for all packages being used is far to large to post

# create instance of BEAST_UPDATE module
my $bdbi = new BEAST_UPDATE();
$bdbi->init('bes', $blast_type, $blast_db, 'all', "");
my $connector = new AAFCBLAST;


for (my $currjob = $startjob; $currjob <= $endjob; $currjob++) {

  $connector->{'job_id'} = $currjob;

  my $query_ids = $connector->getJobQueryIds();

  foreach my $query_id (split ",", $query_ids) {


    print "Getting query result: $query_id of job: ".$connector->{'job
+_id'}."\n";
    # print $query_id. "\n";;
    my $result = $connector->getQueryResult($query_id);

    my $FH;
    open $FH, "<", \$result;

    my $searchio = new Bio::SearchIO(-format => "blast",
                                     -fh => $FH);

    while (my $o_blast = $searchio->next_result()) {
      my $clone_id = $o_blast->query_name();

      my $statement = $bdbi->form_push_SQL ($o_blast, $clone_id, 5);
      print $statement . "\n";
      #$bdbi->update($statement);
    }

    $FH->close;

  }

}
[download]

thanks for the help, Wayne

Well from my GUI experience, memory gains usually comes from creating and destroying objects, and the objects don't have a "zero-ref-count". When that happens, Perl keeps the object in memory because there are still some variables connected to it. It's a hassle.

So you are creating 3 new objects in an interdependent manner, and it's not surprising that Perl dosn't release the memory for reuse.

The first thing to try is to only create 1 of each object, and reuse them over and over, by reconfiguring them with new data on each run. (And undef'ing the previous data)

A second method is to try and reuse the namespace by creating a set of global variables to hold the object, then reusing it. Like:

#globals
my $bdbi;
my $connector;
my $searchio

#then later in the sub, don't declare them as lexical,
#but reuse the global space

sub{

$bdbi = new BEAST_UPDATE();
..........
$connector = new AAFCBLAST;
....
$searchio = new Bio::SearchIO();

.....
}
[download]

A third thing to try, is just to try and undef everything before you leave the sub. You may need to undef everything related to the objects, and do it before you undef the objects. And since the objects are interrelated, undef the objects in the right order.

The first method is probably your best bet, but the others are sometimes "quick fixes".

Also see Re: Is there a penalty for namespace use? and Memory leaks and reference counting within Perl. and the FAQ item

7.27: How do I clear a package?

    Use this code, provided by Mark-Jason Dominus:

        sub scrub_package {
            no strict 'refs';
            my $pack = shift;
            die "Shouldn't delete main package"
                if $pack eq "" || $pack eq "main";
            my $stash = *{$pack . '::'}{HASH};
            my $name;
            foreach $name (keys %$stash) {
                my $fullname = $pack . '::' . $name;
                # Get rid of everything with that name.
                undef $$fullname;
                undef @$fullname;
                undef %$fullname;
                undef &$fullname;
                undef *$fullname;
            }
        }

    Or, if you're using a recent release of Perl, you can just use the
    Symbol::delete_package() function instead.
[download]

---

I'm not really a human, but I play one on earth. flash japh

That's not narrow enough for me to help. I don't have (and don't want) the source to BEAST_UPDATE, Bio::Search, AAFCBLAST, etc. If you're right the leak could be any of them! I'm sorry to say it, but you've still got a lot of work to do.

-sam