Memory leak while printing ?? Don't think so....

baxy77bax has asked for the wisdom of the Perl Monks concerning the following question:

well I have this odd problem with memory leak but i don't see where does the script go wrong. So I inherited some script that basically transforms data from some output given in rows to another output where this data is written in columns. And so after some heavy-duty transforming you finally and with an array like this:

so as you can notice i used Data dumper to get the output. Now the remainder of the program goes as follows:

my $msm = 1;
foreach my $msmtable (@array1){
  open (OUT,">","$ARGV[1].$msm.msm.dat") || die "$!";
  for (my $x = 0;$x <$max[0];$x++){
    foreach my $row (@{$array1[$msmtable]}){
     print OUT "$array1[$msmtable][$row][$x],";
    }
  print OUT  "\n";
  }
  $msm++;
  close OUT;
}
[download]

and it works ok when i don't have the :

print OUT "$array1[$msmtable][$row][$x],";
[download]

line. But when this line is included memory consumption just goes sky-high and finally the system collapses. Now I don't believe that this could be due to printing, I mean, I never had this problem before. So what I'm asking is for suggestions where this could go wrong ?? can the actual data in the array have something to do with it ?? If yes why then the Data::Dumper does not show the same behavior ?? I need help with this !

Cheers

Baxy

Comment on Memory leak while printing ?? Don't think so.... Select or Download Code

Replies are listed 'Best First'.
Re: Memory leak while printing ?? Don't think so.... by ikegami (Patriarch) on Aug 01, 2011 at 09:50 UTC
If `$array1[$msmtable][$row]` doesn't exist, then you'd be expanding an array or two to autovivify that element. `>perl -MData::Dumper -E"my $x = qq{$a[4][2][8]}; print Dumper \@a;" $VAR1 = [ undef, undef, undef, undef, [ undef, undef, [] ] ];` [download] If `$msmtable` or `$row` is very large, then you'd be allocating a very large array. PS — "Memory leak" does not mean "large memory usage".	[reply] [d/l] [select]
Re^2: Memory leak while printing ?? Don't think so.... by ikegami (Patriarch) on Aug 01, 2011 at 09:57 UTC
Looking at your code more carefully, `foreach my $row (@{$array1[$msmtable]}){ print OUT "$array1[$msmtable][$row][$x],"; }` [download] I see that `$row` is a reference to an array, yet you're using that reference as an index. There's the large number we were looking for. `>perl -E"say 0+[];" 6994484` [download] You want: `foreach my $row (@{$array1[$msmtable]}){ print OUT "$row->[$x],"; }` [download]	[reply] [d/l] [select]
Re^3: Memory leak while printing ?? Don't think so.... by baxy77bax (Deacon) on Aug 01, 2011 at 10:06 UTC
Yes you are correct in both, this means that i'm done for today, because i sincerely did not see the second one (need a break)!! Thank you and ++ for prompt and helpful reply !! baxy	[reply]
Re: Memory leak while printing ?? Don't think so.... by jethro (Monsignor) on Aug 01, 2011 at 10:12 UTC
Your memory problem probably comes from the fact that perl has a feature called autovivication: `perl -e 'use Data::Dumper; print $f[5][4][3]; print Dumper(\@f);' $VAR1 = [ undef, undef, undef, undef, undef, [ undef, undef, undef, undef, [] ] ];` [download] Note that a lot of intermediate undef values are generated when all I do is access a value. It doesn't matter for the innermost array as you can see, but if your $row is some really high value, you can fill your memory quite fast. If you don't want that, you could use a hash instead. Or maybe you just have a bug so that $row or $msmtable has a unnaturally high value	[reply] [d/l]