memory leak

camenix has asked for the wisdom of the Perl Monks concerning the following question:

greetings wise monks..
I have a program with an evil memory leak. It may caused by Text::Template Module,but I don't know how to get rid of it.

use Text::Template;
my $linehash={'a'=>'b','c'=>'d'...};
my $Template=Text::Template->new(TYPE=>'FILE',SOURCE=>'D:\sample.xml');
my $Text=$Template->fill_in(HASH=>$linehash);
# memory used increase 2000k,
undef $Template;
#the memory did not released

why?and how to resolved it?

thanks.

I Run PXPerl 5.8.7-6 at WindowsXP SP2,Table has 50000 recode,it cost 1000k every record.
This translate table's record to XML File.

#!perl -w
use strict;

use File::Basename;
use File::Spec;
use Data::Dump; 

use Text::Template;
use Win32::OLE;
use Adoscan;

my $SourceDataConnectionString;
$SourceDataConnectionString='Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security
Info=False;Initial Catalog=GF_GG;Data Source=BEEB';
my $zc='select * from T_2001B0clcp';

#------------------ Main ----------------
print "Begin...at ".localtime()."\n";
# Begin to Convert SQL's Data to text file 
Db2Text(\&Action,$SourceDataConnectionString,$zc);
print "$zc Complete! at".localtime()."\n";

# Begin to transefer Data's index to web's database

}#---------------- End Main --------------

sub Db2Text
{
    my $Action=shift;
    my $DataConnectionString=shift;
    my $string=shift;
    
    # Cycle DataBase TABLE
    Adoscan::scandb($Action,$DataConnectionString,$string);
}

sub Action
{
    my $linehash=shift;
    if( ! $linehash )
    {
         print "Not Define Function! parameter is $linehash \n";
         return;
    }
    my $mark=GetTemplateFile($linehash);       
    my $Template=Text::Template->new(TYPE=>'FILE',SOURCE=>$mark);

my $Text=$Template->fill_in(HASH=>$linehash);
# Do other things
undef $Template;

    return;
}

File:Adoscan # It is my personal lib

package Adoscan;
use strict;
use Data::Dump;
use Win32::OLE::Lite;
use Win32::OLE;
use Win32::OLE::Const;

# despite you do noting,it cost 4k every cycle
sub scandb
{
	my $function=shift;
	my $oledbstring=shift;
	my $commandstring=shift;
	my $conn = Win32::OLE->new('ADODB.Connection'); 
	# my $RS = Win32::OLE->new('ADODB.Recordset');
	$conn->Open($oledbstring);
	if (Win32::OLE->LastError()){             
		print "This didn't go well: ", Win32::OLE->LastError(), "\n";
	}
	my $rst=$conn->Execute($commandstring); 
	if (Win32::OLE->LastError()){             
		print "This didn't go well: ", Win32::OLE->LastError(), "\n";
	}
    # my $rows=[];
	while(!($rst->{EOF}))
	{
		my $currentrow={};
        my $count = $rst->Fields->{Count};
        for(my $i=0;$i<$count;$i++)
	    {
            	my $columnname=$rst->Fields($i)->{Name};
			    my $columnvalue=$rst->Fields($i)->{Value};
			    if(defined($columnvalue))
		    	{
		    		$columnvalue=~s/^\s+//;
		    		$columnvalue=~s/\s+$//;
		    	}
		    	else
		    	{
		    		#$null||($columnvalue='');
		    	}
                $currentrow->{$columnname}=$columnvalue;
		}
		#push(@$rows,$currentrow);
		($debug eq '1')&&(print (caller().$currentrow."\n"));
		&$function($currentrow);
		$rst->MoveNext;
	}
	$conn->Close;
	#return $rows;
}

1;

#END

Comment on memory leak

Replies are listed 'Best First'.
Re: memory leak by Celada (Monk) on Dec 11, 2005 at 18:21 UTC
`# memory used increase 2000k, undef $Template; #the memory did not released` [download] How are you determining the memory usage here? If you are using a tool like `ps` or `top` to view the memory usage of the perl process, then you will get this result. When perl requests memory from the operating system via `malloc`, this memory is not returned until the process dies. When perl is finished with the memory is is made available (with `free`) in the heap so that further memory allocations via `malloc` will use it again until it is exhausted (at which time it will go to the operating system again for more) but it is not returned to the operating system. Read more... (1200 Bytes) If you think instead that the memory occupied by `$Template` has not been freed by perl, then you must assume that someone is still holding a reference to it. That reference could be in a package variable or a toplevel-scoped lexical in the `Text::Template` class for example. You'd have to check the implementation. But I don't think this is what's going on here.	[reply] [d/l] [select]
Re^2: memory leak by camenix (Acolyte) on Dec 12, 2005 at 02:59 UTC
Saddly to hear that I had to check the code.To avoid consuming time,may be I should use fork to run Text::Template. Thanks for your help.	[reply]
Re: memory leak by BrowserUk (Patriarch) on Dec 12, 2005 at 05:20 UTC
I'm guessing that your file sample.xml, is around 300MB in size? Text::Template appears to require around 7 times the filesize in memory, to perform it's manipulations. This will obviously vary with the complexity and number of substitutions in the template, amongst other things. That's not a bug or a leak, just what it takes to do the job. There is a work-around for an early Perl bug in T::Ts file handling that means that the file data gets temporarily duplicated on input, though this is swamped by memory requirements of later processing. A suggestion. Read in the template file yourself, in a few smaller chunks and pass them to T::T as strings. It will require you to ensure that you don't break the file halfway through a piece of T::T markup, but that shouldn't be too onerous. As far as T::T is concerned, everything outside of it's tags is just data. It will preserve it as is, but otherwise not inspect it, so it doesn't matter if you pass in broken chunks of XML, it will still process fine. This way you can read and write chunks serially and reassemble them in your own program space or another file, and you be able to reduce the peak memory consumption to whatever level fits your needs. It might even process more quickly as you will be reusing memory already allocated to the process for the second and subsequent chunks, rather than getting more and more from the OS. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^2: memory leak by camenix (Acolyte) on Feb 08, 2006 at 12:10 UTC
It is longtime for me to review my question,I find above answer. I think I should write something about it: My sample template file(xml) is about 10kB,and my out put file is about 12kB.Each xml file contain 30 variable to replace,and my database's table contain 44 column(about 2000char and 4 text type column).My program cycle a table to create html files,each record create a file from 12 template files. I think your suggestion is correct,because my file has some characters coding by UTF-8 or GB2312.Maybe Text::Template can not treate it correctly.I didn't try to confirm it. To avoid hacking code of Text::Template,I wrote this code to instead Text::Template: Read more... (906 Bytes) It takes 30 minutes to cycle 10000 record.Because I don't need to create html file online,that's enough. Thanks for your help.	[reply]