In general, Class::DBI (or any other O/R mapper) is going to have poor performance when used for bulk loading. To get really good performance on that stuff, you need to use your database's bulk loading tools.

The app walks through a database, pulling "active" rows (rows needing work). It pulls some related info, does a computation, then passes the CDBI objects off to TT for rendering as an output file. As the file is large, rather than pass a big array of Foo to TT, each is passed one by one, and TT appends the result to an open file handle.

Apologies for the obfu names in the code. Here's the loop that does all the work and takes all the time:

# obfu'd code, with some addntl comments


# NB: there are many many of these
my $it = Zmain::Foo->search( client => $client, status => 'active' ); 

    while ( my $Foo = $it->next ) {

# NB this is a master table of Things.  A Foo is a specific kind of Th
+ing.  The master Thing table keeps ids and common info straight.

        my $thing = Zmain::Thing->retrieve($Foo->Foo); 

# NB not sure if this assert wastes time stringifying thing for the fa
+ilure message if the assertion passes?        

        assert( $thing, "found thing=" . $thing->thing );

    my $info        = $Foo->Bar->info; 

        my $computed = $tm->compute( $info, $thing );

        $tt->process(
            'template.tt',
            { Foo => $Foo, computed => $computed },

# just handle one at a time

            sub { print $fh shift } 
        );

}
[download]