in reply to std dev calculations slow over time

I don't know if this is your main problem, but performance for SQLite is poor when doing multiple inserts (see the performance notes in DBD::SQLite). Wrap all of those inserts in a transaction and commit them all at once. I see you have some commit calls in there, but I didn't see where you set up a transaction.

Also, if you think the database might be the culprit, you can profile it with DBI::Profile. I give some examples of that in my Profiling chapter in Mastering Perl. You might also try some of the other profiling techniques to identify other slow parts.

Good luck :)

--
brian d foy <brian@stonehenge.com>
Subscribe to The Perl Review
  • Comment on Re: std dev calculations slow over time

Replies are listed 'Best First'.
Re^2: std dev calculations slow over time
by punkish (Priest) on Oct 22, 2006 at 19:02 UTC
    brian d foy doth spake

    >I see you have some commit calls in there, but
    >I didn't see where you set up a transaction. 

      my $dbh = DBI->connect(
    "dbi:SQLite:$dsn", '','', { RaiseError => 1, AutoCommit => 0 }
  ) or die "$DBI::errstr\n";

    [download]

    I also used Devel::Profiler to do some investigating, but dprofpp tmon.out kept on croaking on me with 

    >dprofpp tmon.out
Modification of non-creatable array value attempted, subscript -1 at C
+:\Perl\bin/dprofpp.bat line 717, <fh> line 1250.

    [download]

    so I gave up that route because I really didn't know wtf that was all about.

    --

    when small people start casting long shadows, it is time to go to bed
[reply]
[d/l]
[select]
      Transactions (in any database support by DBI) are handled with the $dbh->begin_work and $dbh->commit methods. See the "transactions" section of DBI.pm.
[reply]
      my $dbh = DBI->connect( "dbi:SQLite:$dsn", '','',
      { RaiseError => 1, AutoCommit => 0 } )
      or die "$DBI::errstr\n";
      I found with Win32 ActivePerl that explicitly setting $dbh->{AutoCommit} = 0; here helps.
[reply]

In Section Seekers of Perl Wisdom