You don't mention whether your application runs under CGI or a persistent framework such as mod_perl, FastCGI or PersistentPerl.

If you care about performance, I assume you're avoiding CGI so you can reuse database connections and other things that take some time to initialise. Template Toolkit objects are such things: calling Template->new() for each request will make your application run more slowly.

Template Toolkit is used on plenty of high volume Web sites: it's certainly possible to have it run quickly. Unless your application is very simple, TT shouldn't take up 60% of the application's run time.

Thanks for the tips, but the bottleneck turned out to be Template Toolkit. It's taking up an unbelievable 60% of the total invocation time. I'm actually pretty shocked

That's high. Are you:

• Only creating the Template instance once (I'm assuming you're using mod_perl)?
• Caching compiled templates (take a look at the COMPILE_EXT and COMPILE_DIR configuration options)?

I am creating the template instance only once, but I wasn't caching the compiled templates. I'm doing it now, and it cut the response time by 25%! Thanks!

If you're not using some of the more advanced features of TT, perhaps you can switch to a lighter-weight Templating engine? We use HTML::Template, and it does the job for us. Obviously it has a lot less features. I can't guarantee that it's faster, though. Perhaps you can determine what features you MUST-HAVE in your templating system, then install all the ones that meet your needs, and do a comparison of their performance? If you do this, report your findings, since I'm sure others would be interested in the results.

• 60% of what? Is it 10 seconds? 1 minute? 1 second?
• Are you using mod_perl? That one single item will often cut 50% of your response time. And, it's a change that's transparent to the CGI scripts.¹
• Are you using a ton of nested BLOCK directives?
• Are you doing things like

  [% foo = $bar.baz %]
  [% baz = qux.$foo %]
  [download]

  That does a lot of eval work behind the scenes, which can be expensive.

• How deep are your nested loops? Nesting loops don't scale linearly, in any templating system. HTML::Template, which is arguably the most efficient commonly used templating system has serious performance problems with loops nested 3+ deep.
• Can you compile and/or cache the output from some of your templates?
• Are you making calls to the database in your templates using the DBI plugin?

1. Assuming, of course, you used sane coding standards. Persistence can be a bitch if you're converting the first CGI script you ever wrote to run under MP::Registry.

Well...

• Approx 1.25 seconds; caching the compiled templates took it down to approx 1 second.
• yes
• no
• no, I'm use .item when working with nested hashes
• Now that's something I didn't know. The main data section has nested loops four-deep. Column order is defined in a config file, so we do something like this:

      [% FOREACH set IN [data_header.top_priority,
                         data_header.middle_priority] %]
      [% FOREACH fieldno IN set %]
      [% field = data_header.field.slice(fieldno, fieldno).0 %]
      [% value = order.item(make_key(field.title))
      ...
  [download]

  Which probably isn't very nice to the TT processor.
• I wasn't, but I am now.
• No, calls to the database are made through callback functions.

No, calls to the database are made through callback functions.

I guess I wasn't being specific enough. In apps using H::T, all database activity has to be completed before H::T can be used. However, TT allows you to make database calls while TT is running, and it sounds like you're using that functionality.

One way, obviously, is to use the DBI plugin. Another way is use callback functions, as you're using. I'll bet 1000XP that if you were to benchmark how much of the template time is being spent in those callbacks, it'll be at least 50%.

Also, I'm not a TT expert, but [% field = data_header.field.slice(fieldno,fieldno).0 %] looks horribly inefficient. Woudn't [% field = data_header.field.$fieldno %] suffice?

Update: Fixed to jryan's reply.

------
We are the carpenters and bricklayers of the Information Age.

Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

I shouldn't have to say this, but any code, unless otherwise stated, is untested