anjiro has asked for the wisdom of the Perl Monks concerning the following question:

So I've got a big pile of tables in a mySQL database and I want to get a list of only the tables that have data; that is, I want to list all tables but those with zero rows. The best way I've been able to come up with to do this is the relatively ugly following code. There must be a better way!
#Assuming $sth has just been executed with a "show tables" query while(@tmp = $sth->fetchrow_array()) { my $tablename = $tmp[0]; my $sth2 = $dbh->prepare("select count(*) from $tablename"); $sth2->execute(); $numrows = ($sth2->fetchrow_array())[0]; if($numrows > 0) { push @tables, $tablename; } }
The whole having to create and execute another statement just seems... excessive. Non-perl like. It's especially pesky since DBI won't let me do something like this, which at least would allow me to call "prepare" only once:
my $sth2 = $dbh->prepare("select count(*) from ?"); $sth2->execute($tablename);
Apologies if this is more an SQL question than a perl question... I've poked around the DBI docs and haven't seen anything obvious.

Replies are listed 'Best First'.
Re: DBI: Better way to find number of rows in a table?
by VSarkiss (Monsignor) on Jun 04, 2003 at 03:11 UTC

    Not only is it not a Perl question, or even a SQL question, it's a platform-specific MySQL question. In other words, the answer is different for Oracle, DB2, SQL Server, etc. All of these have different catalogs, so the "improved" query would be different. Even knowing the catalog structure won't always help because most databases don't allow variables in the FROM clause: you end up running the same query for each table anyway.

    There is a better way to ask if a table has rows, but I'm not sure if MySQL supports it: the EXISTS clause. In Sybase, for example, you could say:

    if exists (select 1 from my_table) print 'my_table has at least one row'
    Otherwise, it's kind of a waste to count all the rows just to see if the table's empty.

(jeffa) Re: DBI: Better way to find number of rows in a table?
by jeffa (Bishop) on Jun 04, 2003 at 04:43 UTC
    Sounds to me like you are only going to run this script one time. Once you get your list, you move on, throw this one away. If that's the case, then who cares how you got the list, as long as the list is correct? I do appreciate wanting to find a better way of doing something, but sometimes you should just move on.

    Off the top of my head, however, i would try to utilize map or grep. Sometimes they are more elegant when it comes to populating a list with data. Also, the select* functions from DBI allow you to combine prepare, execute, and fetch functions into one call. Something like:

    my @empty = grep { $dbh->selectcol_arrayref("select count(*) from $_")->[0] == 0 } $dbh->tables;
    might suffice ... dunno about performance, but then again, if you only run this once and the amount of time it takes to execute is acceptable, why change it? ;)

    UPDATE:
    changed to $dbh->tables per gmax's suggestion, but now it looks just like Zaxo's!! :D (i orginally used @{ $dbh->selectcol_arrayref("show tables") })

    jeffa

    L-LL-L--L-LL-L--L-LL-L--
    -R--R-RR-R--R-RR-R--R-RR
    B--B--B--B--B--B--B--B--
    H---H---H---H---H---H---
    (the triplet paradiddle with high-hat)
    
Re: DBI: Better way to find number of rows in a table?
by jaa (Friar) on Jun 04, 2003 at 08:13 UTC
    You can do this in one query in MySQL since version 3.23 - try

    SHOW TABLE STATUS

    It also supports LIKE 'MYTABLE%' wildcards.

    Regards

    Jeff

Re: DBI: Better way to find number of rows in a table?
by Zaxo (Archbishop) on Jun 04, 2003 at 05:14 UTC

    If your DBI and the MySQL bits are recent, you can do this:

    # $dbi handle is available my @non_empties = grep { # almost like jeffa's now $dbh->selectcol_arrayref("select count(*) from $_")->[0] } $dbh->tables;

    Update: rnahi is correct, $sth->rows won't do. Repaired

    After Compline,
    Zaxo

      This won't work as expected.

      $sth->rows returns the number of affected rows, and since in this case there are none, (COUNT is not a row affecting command) it will return -1, thus making TRUE all the items in the array. Therefore grep will return all the tables, even the empty ones.

Re: DBI: Better way to find number of rows in a table?
by hmerrill (Friar) on Jun 04, 2003 at 12:38 UTC
    Nothing wrong with your code - it gets the job done, it's easy to understand - it's not that big. "jeffa" had a good example using the perl "grep" function, but I find those long one or two liners harder to understand than something simple (with a few more lines) like you've done here.

    As "jeffa" (I think) pointed out, you could probably shorten your code by replacing the prepare, execute, and fetch with a selectrow_arrayref - from 'perldoc DBI':
    "selectrow_arrayref" $ary_ref = $dbh->selectrow_arrayref($statement); $ary_ref = $dbh->selectrow_arrayref($statement, \%attr); $ary_ref = $dbh->selectrow_arrayref($statement, \%attr, @ +bind_values); This utility method combines "prepare", "execute" and "fetchrow_arrayref" into a single call. It returns the firs +t row of data from the statement. The $statement parameter can be a + previ- ously prepared statement handle, in which case the "prepare +" is skipped. + If any method fails, and "RaiseError" is not set, "selectro +w_array" will return undef.
    HTH.
Re: DBI: Better way to find number of rows in a table?
by tedrek (Pilgrim) on Jun 04, 2003 at 06:23 UTC

    just as a note under mySQL I would use limit rather than count. eg.

    my $sth = $dbh->prepare("select * from $table limit 1"); $sth->execute(); print "Theres at least 1 row" if $sth->rows();

    That way you don't have to look through the whole table to create the result ( although it's possible that is optimized)

    note: untested

    Update: mahi's note below shows just why one should be careful about premature optimization. heh.

      Your method will work, but not the way you say.

      In MySQL, COUNT(*) (without a WHERE clause) is optimized so that it will get the result from the description table, without physically counting the records.

      Thus, your query will be slower than using COUNT. ;)

      jeffa's method is the fastest you can get in this case.

Re: DBI: Better way to find number of rows in a table?
by Itatsumaki (Friar) on Jun 04, 2003 at 16:51 UTC

    Indeed there isn't any way to dynamically set table names in DBI, or indeed in most databases. The reason why comes down to what a ->prepare() is actually doing. (Note: this is somewhat platform specific, so YMMV here). The prepare tells the database to get the SQL ready and parsed so that it can be executed with a variety of parameters in the where clause. This is an efficiency issue for the DB, and there is no way to dynamically avoid it.

    So, what can you do?

    • To get the row-count you can either do the count(*) (as you mentioned) or retrieve all rows and count them
    • To get by the dynamic table problem you can write a simple sub (see below) that takes the table-name and returns you the SQL.
    • That sub can be linked with the cute DBI method $dbh->selectrow_array() which can remove the need for the prepare/execute combination and save you a few lines of code

    sub make_sql($) { return "SELECT COUNT(*) FROM $_[0]"; } if ($dbh->selectrow_array(make_sql($tablename))) { # do stuff }

    A final note: something seems buggy to me in using the selectrow_array method directly in the if condition, but I can't put my finger on what it is. You might want to break it into a separate variable $count = $dbh->selectrow_array(make_sql($tablename));

    Hth!
    -Tats

    Update: I just realized that my response is nearly identical to hmerril's just above....