clearcache has asked for the wisdom of the Perl Monks concerning the following question:

Hi all - this is my first post, so I hope I don't break any etiquette rules ;)

I'm writing a subroutine that seems to be causing my memory usage to spike...It is part of a larger program that is taking data (and metadata) out of a legacy data collection application and prepping it for import into a relational database.

I have narrowed it down to the following code (acutally, I have narrowed it down to one line, the line that dereferences $rDTX...I can't see why dereferencing something already in memory should cause mem usage to spike, but when that line is commented out, all is good.) It's unfortunate that it's such a large program, so I can't post the entire thing here, but any insight would be greatly appreciated. 
 open(OUT,">$outfile") || die "Can not open $outfile for writing ($!)\
+n";
 my $ctr = 0;
 for my $key (sort keys %$rDDT) {
   $ctr++;
   for my $quest (@{$$rMAP{'Order'}}) {
     my $txqu = uc($quest);
     for my $iter (@{$$rMAP{$quest}{'Iterations'}}) {
       my $txit = sprintf "%03d", $iter;
       my $qtyp = $$rMAP{$quest}{$iter}{'QType'};
       my $card = sprintf "%02d", $$rMAP{$quest}{$iter}{'QCard'};
       my $scol = $$rMAP{$quest}{$iter}{'QSCol'};
       my $ecol = $$rMAP{$quest}{$iter}{'QECol'};
       my $spos = $scol - 1;
       my $flen = $ecol - $spos;
       my $dat  = $$rDDT{$key}{$card}{'data'};
       my $mp   = $$rDDT{$key}{$card}{'multis'};
       my $rtxt = $$rDTX{$key}{$txqu}{$txit}{'ResponseText'}; # this i
+s the line that is causing me problems
       if ($rtxt) { print "$rtxt\n"; undef $rtxt }
       my $data = ();
       $data = "$key\t".uc($quest)."_$iter\t";
       $data .= getData($dat,
                        $mp,
                        $rQDI,
                        $card,
                        $scol,
                        $spos,
                        $ecol,
                        $rtxt
                       );
        writeReport($data, $rOptions, \*OUT);
      }
    }
    progressDisp($ctr);
  }
  print "\n\t$ctr respondents profiled\n\n";
  close(OUT);
  print "leaving makeProfiles()\n\n";
}

[download]
Thanks!

Edit kudra, 2001-10-31 Changed title

Replies are listed 'Best First'.
Re: Memory problems...
by chromatic (Archbishop) on Oct 31, 2001 at 10:17 UTC
    That doesn't make immediate sense to me. The only thing that comes to mind is the possibility that you're looping thousands and thousands of times, and there's nothing in $rtxt for most of those times. Hash autovivification could be creating new memory structures for you. Something like this could help: 
    if (exists $rDTX->{$key}{$txqu}{$txit}{ResponseText}) {
    print $rDTX->{$key}{$txqu}{$txit}{ResponseText}, "\n";
}

    [download]
    It's a little longer to type, but exists skips the autovivification. Depending on what's defined in $rDTX and where, you may have to see if more keys exist (like $txit, $txqu, and $key). It's hard to tell without seeing example data, though.
[reply]
[d/l]
      Yes, that is the case - $rDTX may not always contain info for {$key}{$txqu}{$txit}{ResponseText}...and yes, I am looping thousands and thousands of times.

      I wasn't aware of "hash autovivification"...and, I know - it is very hard to tell without seeing example data...I appreciate the input.
[reply]
(tye)Re: Memory problems...
by tye (Sage) on Oct 31, 2001 at 10:28 UTC

    There is or was a "bug" in that "my" variables created inside of a loop were not free()d until after the last iteration of the loop. You could try moving all of those my declarations to outside of the loop or add a bare block inside of the loop such that all of the my declarations are inside of that. Or you could try upgrading to a newer version of Perl.

    Update: I don't know what version(s) of Perl this problem affects.

            - tye (but my friends call me "Tye")
[reply]
      cati2:: perl -v

This is perl, v5.6.1 built for sun4-solaris

      [download]
      was it broken in this version?
[reply]
[d/l]

Back to Seekers of Perl Wisdom