bman has asked for the wisdom of the Perl Monks concerning the following question:

I come to your gates to find out where my bottle neck is. I am connecting to MySQL database (holding at the current time about 400,000 records and growin with every day) via web interface and PERL, of course. My problems is that to return a query it takes between 30 seconds to 2.10 minutes.

I know that database is not an issue because I can query in database itself in a jiffy. No slow downs. When I look into MySQL stats, I see many "slow queries" entries. I presume that this is from the web.

This is my piece of code: snip...]

if ($Choice eq 't_datein') { if ($Choice2 eq 't_datein') { $query .= " ORDER BY $Choice $SortOrder, t_timein $SortOrder, +$Choice2, t_timein $SortOrder"; } else { $query .= " ORDER BY $Choice $SortOrder, t_timein $SortOrder, +$Choice2"; } } else { $query .= " ORDER BY $Choice $SortOrder, $Choice2"; } if ($Top ne 'All') { $query .= " LIMIT $Top"; } $sth = $dbh->prepare($query); $sth->execute();
and

snip..]

if (($Top eq 'All') && (!$SearchString)) { $total = $sth->fetchrow_array (); $total = "I'm confused" if !defined ($total); print "</table style=\"border: 1 solid #000080\" align='center'>\n +"; print "<p>\n"; print "<h4>Query has returned $total number of records.</h4>\n"; print "<hr width=\"98%\" align='left'>Please, narrow down your cry +teria or make sure that if"; print "<b><font color='red'> ALL is selected</b></font>,<br>"; print "a string is provided in the query field as well!<p>\n"; } else { while (($t_datein, $t_timein, $t_dateout, $t_timeout, $t_port, $t_prot, $t_where, $t_from, $t_pkt_out, $t_pkt_in, $t_byte_in, $t_byte_out, $t_id) = $sth->fetchrow_ar +ray ()) { $skip = $i % 2; $i++; $bytes = $t_byte_out + $bytes; if ($t_where =~ /(adult|sex|oral|porn|xxx)/i) #you get an idea + { print "<tr bgcolor=\"#FF9999\">\n"; } elsif ($skip eq 0 ) { print "<tr bgcolor=\"#CCFFCC\">\n"; } else { print "<tr>\n"; } print " <td align='right' bgcolor='black' align='top'><font co +lor='white'><b>$i</b></font></td>\n"; print " <td align='right' align='top'>$t_from</td><td align +='right' align='top'>"; if ($t_where =~ /.*\.[a-z][a-z][a-z]?$/i) { print "<a href=\"http://$t_where\">$t_where</a>"; } else { print "$t_where"; } print "</td>\n"; print " <td align='right' align='top'>$t_datein $t_timein</ +td><td align='right' align='top'>$t_dateout $t_timeout</td>\n"; print "<td align='right' align='top'>$t_byte_out</td>\n"; } $bytes = ($bytes / 1024) / 1024; } ($sth)->finish(); ($dbh)->disconnect();
That's pretty much the core what I have been doing. I think that the problem is my code. How can I speed up those queries (for example, a simple 'select count(*) from table' shouldn't take 30 seconds).

I really appreciate your words of wisdom. :-)

Replies are listed 'Best First'.
Re: Speed blues
by Fastolfe (Vicar) on Oct 25, 2000 at 20:03 UTC
    Perhaps this is obvious, but do you have $| set to 1 in your code? Stdio buffering might cause things to be sent to your browser slowly from the script.

    If that's OK, perhaps you should inject some debugging code in key places in your code, printing the time (perhaps making use of Time::HiRes) at each point. See if you can use code like that to find out which part of your script is causing the delay, and narrow it down to the exact event. If you have trouble doing this, see if you can break your script down into the simplest possible case that demonstrates the delay you're seeing and respond to this post with it.

    I can't see anything obviously wrong with your code, bot perhaps someone else might... I know it is kinda weird to see ($sth)->finish(); when $sth->finish; would work exactly the same and looks a little cleaner. :)

    Good luck.

Re: Speed blues
by wardk (Deacon) on Oct 25, 2000 at 20:49 UTC

    you might check out: MySQL Benchmark Suite

    I notice they have a benchmark listed that a 2mil row indexed read shows a time of 367 seconds.

    I suspect that you might be reaching the limits of mySQL, or at least the limits that guarantee "acceptable" response times.

RE: Speed blues
by Jonathan (Curate) on Oct 25, 2000 at 20:33 UTC
    There's not much to go on here, the codes certainly no worse than my own.
    How do you know the database is not the choke point? (they usually are :-). I don't know MySql (its the only major DB I've never used) But Count(*) and Order by often entail table scanning and temp tables. How is your database set up, are all your most active tables on the same disk? Is this not a concurrency issue? A query in isolation can run fine but when 200 people are trying to do the same thing at the same time...

    Dunno give up.
    400,000 records growing all the time are you sure this isn't a porn site ;-)

    "We are all prompted by the same motives, all deceived by the same fallacies, all animated by hope, obstructed by danger, entangled by desire, and seduced by pleasure." - Samuel Johnson
Re: Speed blues
by Maclir (Curate) on Oct 26, 2000 at 01:34 UTC
    Hmmmm - "order by" . . . . . 400,000 records . . . tell me - what indexes are on the table(s) you are doing your SELECT on? I have not used MySQL, but I suspect its query optimiser is not as flash as with (say), Oracle, Informix or DB/2.

    You may need to look at your physical data base design, to make sure your queries are not scanning through the whole table multiple times.

Re: Speed blues
by cianoz (Friar) on Oct 26, 2000 at 04:24 UTC
    be shure you have an index on every column you use on a WHERE or ORDER BY statement, you should also replicate your query from the command line using EXPLAIN SELECT ... to figure out HOW things are done and WHERE you can optimize your query or table structure.
RE: Speed blues
by lachoy (Parson) on Oct 25, 2000 at 21:14 UTC

    I've found the best thing to do is, after generating the SQL, dump it to a file or to a log somewhere. Once the script is done running, copy-and-paste the SQL into your friendly mysql shell and see what kind of performance you're looking at. Eyeballing is the first step, then doing the tasks that others have mentioned is a good idea. But if your problem is solved after eyeballing, the virtue of laziness comes into play (in a slightly altered form).