in reply to Much slower DBI on RHEL6

Hi,

not an answer to your main question. But a kind of annotation looking at your code:

IMHO you're not using the possibility of prepared statements and bind variables. Look at this code:

my $dbh = $me->{DBHANDLE};
my %labels;

# Get all the different ids
my $sql = 'SELECT DISTINCT id FROM list_index ';
my $sth = $dbh->prepare($sql);
$sth->execute;

my $sql2 = 'select id, description from list_index  where id = ? order
+ by rpt_key desc limit 1';
my $sth2 = $dbh->prepare($sql2);

while(my $hashref = $sth->fetchrow_hashref) {
    $sth2->execute($hashref->{id});
    while(my $hashref2 = $sth2->fetchrow_hashref) {
        $labels{$hashref2->{id}} =  $hashref2->{id} . ' - ' . $hashref
+2->{description};
    }
}
$sth2->finish;
$sth->finish;

return \%labels;

[download]

With this code you gain two things: a) You don't prepare a new sql statement per loop iteration. b) You use bind variables which is almost always better than inserting values like you did it (sql injection). c) It should be measurable faster.

Best regards
McA

Replies are listed 'Best First'.
Re^2: Much slower DBI on RHEL6
by Tux (Canon) on Feb 06, 2014 at 12:44 UTC

    Nice start, but fetching hashrefs is the slowest possible access method. Binding return values might improve a lot

    my $dbh = $me->{DBHANDLE};
my %labels;

# Get all the different ids
my $sth = $dbh->prepare ("select distinct id from list_index");
$sth->execute;
$sth->bind_columns (\my $id);

my $sth2 = $dbh->prepare (qq;
    select description
    from   list_index
    where  id = ?
    order by rpt_key desc;
    # -- I don't know if limit 1 is needed
    );
$sth2->execute (0);
$sth2->bind_columns (\my $desc);

while ($sth->fetch) {
    $sth2->execute ($id);
    while ($sth2->fetch) {
        $labels{$id} = "$id - $desc";
        }
    }
$_->finish for $sth, $sth2;

return \%labels;

    [download]

    Enjoy, Have FUN! H.Merijn
[reply]
[d/l]

      Hi Tux,

      the focus in my proposal was on using bind variables and using a prepared statement more than once. But you're right, when someone is hunting the milliseconds, your approach should be faster.

      Thank you and ++.

      Best regards
      McA

[reply]

        More than milliseconds. Last time I tested on MySQL, it was 5 to 10 times faster. (also this table where higher numbers are better)


        Enjoy, Have FUN! H.Merijn
[reply]

      thank you EVERYBODY for the responses. I didn't mention it before but I had tried optimizing things by using different types of fetches, binding, etc. but nothing seemed to help that much. However, I do have things working much faster now( even faster than it was on RHEL 4)!. I don't know why, but here is what I found thanks to tux's post. I do need the "limit 1" because otherwise, I don't get the most recent description, I get the oldest. However, for whatever reason, when I use desc AND limit 1 things are slloooww. Without "limit 1" things are fast... So, it is much faster for me to iterate through all of the rows of ids just to get the "latest" than it is for me to sort in descending order and get just 1. Hope that makes sense. Thanks again everyone. My code is much more optimized and I'm sure throughout the application there are things can be optimized and more secure so I am going to work on that. So, in short, still don't know why the same exact Perl code, with the same data, is much slower on RHEL 6 but my issue is resolved. So this is the code I ended up using and it is faster and seems to give me the correct results

      my $dbh = $me->{DBHANDLE};
my %labels;

# Get all the different ids
my $sth = $dbh->prepare ("select distinct id from list_index");
$sth->execute;
$sth->bind_columns (\my $id);

my $sth2 = $dbh->prepare (qq;
    select description
    from   list_index
    where  id = ?
    order by rpt_key;
    );
$sth2->execute (0);
$sth2->bind_columns (\my $desc);

while ($sth->fetch) {
    $sth2->execute ($id);
    while ($sth2->fetch) {
        $labels{$id} = "$id - $desc";
        }
    }
$_->finish for $sth, $sth2;

return \%labels;

      [download]
[reply]
[d/l]
[reply]

In Section Seekers of Perl Wisdom