in reply to memory leak

I'm guessing that your file sample.xml, is around 300MB in size? Text::Template appears to require around 7 times the filesize in memory, to perform it's manipulations. This will obviously vary with the complexity and number of substitutions in the template, amongst other things. That's not a bug or a leak, just what it takes to do the job.

There is a work-around for an early Perl bug in T::Ts file handling that means that the file data gets temporarily duplicated on input, though this is swamped by memory requirements of later processing.

A suggestion. Read in the template file yourself, in a few smaller chunks and pass them to T::T as strings. It will require you to ensure that you don't break the file halfway through a piece of T::T markup, but that shouldn't be too onerous.

As far as T::T is concerned, everything outside of it's tags is just data. It will preserve it as is, but otherwise not inspect it, so it doesn't matter if you pass in broken chunks of XML, it will still process fine. This way you can read and write chunks serially and reassemble them in your own program space or another file, and you be able to reduce the peak memory consumption to whatever level fits your needs.

It might even process more quickly as you will be reusing memory already allocated to the process for the second and subsequent chunks, rather than getting more and more from the OS.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: memory leak
by camenix (Acolyte) on Feb 08, 2006 at 12:10 UTC
    It is longtime for me to review my question,I find above answer.
    I think I should write something about it:

    My sample template file(xml) is about 10kB,and my out put file is about 12kB.Each xml file contain 30 variable to replace,and my database's table contain 44 column(about 2000char and 4 text type column).My program cycle a table to create html files,each record create a file from 12 template files.

    I think your suggestion is correct,because my file has some characters coding by UTF-8 or GB2312.Maybe Text::Template can not treate it correctly.I didn't try to confirm it.

    To avoid hacking code of Text::Template,I wrote this code to instead Text::Template:
    Read more... (906 Bytes)
    It takes 30 minutes to cycle 10000 record.Because I don't need to create html file online,that's enough.

    Thanks for your help.
[reply]

In Section Seekers of Perl Wisdom