in reply to How to improve MYSQL search performance of perl?

Initial thought is that you need to examine your queries and your table structure and indicies. 500MB isn't all that much if the data is index'd properly. Can you provide some sample queries and schema?
  • Comment on Re: How to improve MYSQL search performance of perl?

Replies are listed 'Best First'.
Re^2: How to improve MYSQL search performance of perl?
by nan (Novice) on Aug 19, 2005 at 15:32 UTC

    Hi,

    As I need to read line by line and search them in the database, I used a subroutine to handle all database work. Below is my code.

    thanks again,

    Nan

      I would take trammell's suggestion a step further and also not recreate the statement handle every time and actually take advantage of the statement handle (and placeholders)-- i think this will have a decent improvement in the performance (amount of gain is probably db-dependent):
      my $dbh = DBI->connect('DBI:mysql:diet', {RaiseError => 1, AutoCommit +=> 0} ) || die "Failed to connect: $DBI::errstr"; my $sth = $dbh->prepare( qq{select topic FROM table1 WHERE uri LIKE ?} + ); search($sth, 'foo'); search($sth, 'bar'); $sth->finish(); $dbh->disconnect(); #disconnect from database; sub search{ my $sth = shift; # require statement handle (this could probably be + a global var instead if desired) my $q = shift; # take search parameter from html <form/> my $found = 0; #initialize category found count; $sth->execute($q); my $rows = $sth->fetchall_arrayref( {} ); printf "%d rows found for '%s'.\n", scalar(@$rows), $q; foreach my $row (@$rows){ printf " Topic: %s\n", &topic($row->{topic}); } }

        Hi,

        Many thanks for the code as I learned a lot from you. But even opened database once, the search is still slow (running 200 queries one by one) and I always recieve Internal Server 500 Error after a long waiting and can't see any results back to screen (before modifying, I could see a few results back as the database open and close everytime). I'm not sure it is due to the configuration of apache server or MySQL database so I'm rebuilding the database now to see if it helps.

        thanks again,

        Nan

      One improvement you can make is to only open your database handle once at the beginning of the script, and reuse that handle instead of recreating it for each query.