Momusian Monks,

In trying to tune my DBI-driven database middleware, I tried what is certainly poor form and got a surprise.

The mysql database has just two columns, col_1 & col_2, both smallint, but over 12-million rows. Neither column can be guaranteed unique values, but combinations of them are definitely unique. So, I created a compound Primary Key.

I want to count how many rows with the same col_2 value have either x or y in col_1.

To benchmark the database performance independently of the Perl script, I telnetted in and did:

select count(*) from theTable A inner join theTable B where (A.col_1 = x and B.col_1 = y and A.col_2 = B.col_2);

which gave me an average response time of ~0.13 sec. (Many thanks to those in the other thread who encouraged me to try various index configurations)

Then, I wondered how much time the join was adding, so I wrote a perl script: 

        $sql = "SELECT col_2 FROM theTable WHERE col_1 = $value_1";
        $sth = $dbh->prepare($sql) or die("Could not prepare!" . $dbh-
+>errstr);
        $sth->execute() or die("Could not execute 1!" . $dbh->errstr);
        while ($data = $sth->fetchrow_array()) {
            push(@col_1, $data);
        }
        $sql = "SELECT col_2 FROM theTable WHERE col_1 = $value_2";
        $sth = $dbh->prepare($sql) or die("Could not prepare!" . $dbh-
+>errstr);
        $sth->execute() or die("Could not execute 2!" . $dbh->errstr);
        while ($data = $sth->fetchrow_array()) {
            push(@col_2, $data);
    }

foreach $e (@col_2) { $union{$e} = 1 }
foreach $e (@col_1) {
    if ( $union{$e} ) { $isect{$e} = 1 }
}
@isect = keys %isect;

[download]
Essentially doing the join by finding the intersection of the two query result arrays myself in Perl. This had an average response time of ~0.10 sec! I expect it will be evn better if I replace the repetitive prepares with placeholders.

Now, I expect to be savaged for moving database functinality to Perl code, but if it works better......

In reply to Basic Perl array intersection faster than mysql query join. by punch_card_don

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.