Creating huge files with the Template Toolkit

holli has asked for the wisdom of the Perl Monks concerning the following question:

In the near future I will have to create huge XML files (along the lines of 4-6 GB (yes, Gigabytes)) from various data sources. I was thinking to use the Template-Toolkit for this task. I played around with it and like the way it works.

However, after some deeper studying of the manual I realized, that all of the templates output gets concatenated to a single string. That of course will cause a huge memory footprint when dealing with output files that big. So I am now thinking about two ways:

Find a way to "flush" (emptying the output string after printing it) the template in time so, I can spare memory.
Use another Templating engine that does not have this limitation.

I would prefer the first option, since I am now familiar with the TT and really like it's syntax and plugin mechanism.

I'd be happy if you share your insights with me.

holli, /regexed monk/

Comment on Creating huge files with the Template Toolkit Download Code

Replies are listed 'Best First'.
Re: Creating huge files with the Template Toolkit by anonymized user 468275 (Curate) on Jul 15, 2005 at 12:35 UTC
The object-oriented nature of TT suggests a possible strategy: - Identify the method that is handling the strings in the way you want to avoid. - Create your own class that inherits from the TT class. - In your own class, override the offending method to give you control over I/O buffering - perhaps also using additional classes such as IO::File and IO::String Hope this helps. -Simon One world, one people	[reply]
Re: Creating huge files with the Template Toolkit by bravenmd (Sexton) on Jul 15, 2005 at 14:09 UTC
I have used the XML::Writer utility from CPAN to create similar sized XML documents. I have never run into any memory issues with it but that does't mean they don't exist. Take a look and see if it may help you. The module is very easy to use and I found great value in it.	[reply]
Re^2: Creating huge files with the Template Toolkit by ruoso (Curate) on Jul 15, 2005 at 17:02 UTC
Indeed. Do you really need to use TT? I really can't think about a reason to use it, since XML::Writer would deal better with special characters, encoding and so on... daniel	[reply]
Re: Creating huge files with the Template Toolkit by thunders (Priest) on Jul 15, 2005 at 14:29 UTC
I found this thread on the mailing list. Click here It sounds like a bit of a hack, basically you alter the $OUTPUT token for the perl code TT generates. In Template::Directive it is defined as: `$OUTPUT = '$output .= ';` You can set it to something like this `$Template::Directive::OUTPUT = 'print ';` On the thread Andy notes that this will break WRAPPER and lists a few other caveats.	[reply] [d/l] [select]
Re^2: Creating huge files with the Template Toolkit by tinita (Parson) on Jun 22, 2006 at 15:57 UTC
can't you just output to a filehandle? (note that I'm not a TT expert, I'm just wondering)	[reply]
Re: Creating huge files with the Template Toolkit by perrin (Chancellor) on Jul 15, 2005 at 12:54 UTC
Someone did this years ago and posted it to the TT list. Check the archives.	[reply]
Re^2: Creating huge files with the Template Toolkit by holli (Abbot) on Jul 15, 2005 at 13:08 UTC
Thanks for the info. ~~I must admit I am absolutely unfamiliar with mailing lists. I am currently googling, but I would appreciate if you could assist me a bit further :)~~ I have found this thread. I will email myk and see if I can get a hand on his patch. holli, /regexed monk/	[reply]
Re: Creating huge files with the Template Toolkit by mikeraz (Friar) on Jul 15, 2005 at 16:26 UTC
Curiosity time. Just what are you doing to create 4GB xml files? Be Appropriate && Follow Your Curiosity	[reply]