Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Creating huge files with the Template Toolkit

by holli (Abbot)
on Jul 15, 2005 at 11:52 UTC ( #475184=perlquestion: print w/replies, xml ) Need Help??

holli has asked for the wisdom of the Perl Monks concerning the following question:

In the near future I will have to create huge XML files (along the lines of 4-6 GB (yes, Gigabytes)) from various data sources. I was thinking to use the Template-Toolkit for this task. I played around with it and like the way it works.

However, after some deeper studying of the manual I realized, that all of the templates output gets concatenated to a single string. That of course will cause a huge memory footprint when dealing with output files that big. So I am now thinking about two ways:
  1. Find a way to "flush" (emptying the output string after printing it) the template in time so, I can spare memory.
  2. Use another Templating engine that does not have this limitation.
I would prefer the first option, since I am now familiar with the TT and really like it's syntax and plugin mechanism.

I'd be happy if you share your insights with me.

holli, /regexed monk/

Replies are listed 'Best First'.
Re: Creating huge files with the Template Toolkit
by anonymized user 468275 (Curate) on Jul 15, 2005 at 12:35 UTC
    The object-oriented nature of TT suggests a possible strategy:

    - Identify the method that is handling the strings in the way you want to avoid.

    - Create your own class that inherits from the TT class.

    - In your own class, override the offending method to give you control over I/O buffering - perhaps also using additional classes such as IO::File and IO::String

    Hope this helps.


    One world, one people

Re: Creating huge files with the Template Toolkit
by bravenmd (Sexton) on Jul 15, 2005 at 14:09 UTC
    I have used the XML::Writer utility from CPAN to create similar sized XML documents. I have never run into any memory issues with it but that does't mean they don't exist. Take a look and see if it may help you. The module is very easy to use and I found great value in it.
      Indeed. Do you *really* need to use TT? I really can't think about a reason to use it, since XML::Writer would deal better with special characters, encoding and so on...
Re: Creating huge files with the Template Toolkit
by thunders (Priest) on Jul 15, 2005 at 14:29 UTC
    I found this thread on the mailing list. Click here

    It sounds like a bit of a hack, basically you alter the $OUTPUT token for the perl code TT generates.

    In Template::Directive it is defined as:
    $OUTPUT    = '$output .= ';

    You can set it to something like this
    $Template::Directive::OUTPUT = 'print ';

    On the thread Andy notes that this will break WRAPPER and lists a few other caveats.

      can't you just output to a filehandle? (note that I'm not a TT expert, I'm just wondering)
Re: Creating huge files with the Template Toolkit
by perrin (Chancellor) on Jul 15, 2005 at 12:54 UTC
    Someone did this years ago and posted it to the TT list. Check the archives.
      Thanks for the info. I must admit I am absolutely unfamiliar with mailing lists. I am currently googling, but I would appreciate if you could assist me a bit further :)

      I have found this thread. I will email myk and see if I can get a hand on his patch.

      holli, /regexed monk/
Re: Creating huge files with the Template Toolkit
by mikeraz (Friar) on Jul 15, 2005 at 16:26 UTC

    Curiosity time. Just what are you doing to create 4GB xml files?

    Be Appropriate && Follow Your Curiosity

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://475184]
Approved by marto
Front-paged by tlm
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (2)
As of 2022-12-04 21:41 GMT
Find Nodes?
    Voting Booth?

    No recent polls found