in reply to Foreach loop takes a long time to process and my report times out before page is rendered

Here are some thoughts I think apply.

One problem, you're using templating which requires that you build a very big data structure in memory, just so you can print it out.

The solution is to NOT build a giant data structure in memory, and print each row right after you retrieve it.

All of these variable seem redundant

my ($key, %col_cbud, %col_encu, %col_fytd, %col_proj, %ctrl_tbl, @loop +_data);
and I can tell without even seeing your template.

You might say but I need all those to smoosh the datasets together.

Well, thats what views are for. If you under utilize your database, you end up replicating a lot of its functionality. Databases are usually very good (well, good enough, depends on the data) at smooshing a bunch of datasets together.

In a recent thread, before I remembered about a special view called a pivot table, I ended up replicating the feature in perl, see Re: Open multiple file handles?. It took a while to write, a lot longer than it would to use a pivot table.

Leverage your database, study the sql, optimize it, its cheaper than reimplementing in perl. I realize its possible you may have good reasons for doing things in perl :)

Now the real trick to speeding up slow, long running programs, CGI or otherwise, is to refuse to run more than one (schedule only one) and cache the results.

For a technique see Watching long processes through CGI (Aug 02)

  • Comment on Re: Foreach loop takes a long time to process and my report times out before page is rendered
  • Download Code