I have a huge (21M records ) table and got sphinx search to get me the id of the searched match for each record. That works excellent and really fast, just nicely :)

Now to be fast in building the web-pages and to be able to answer around 10 pagehits/s and each containing about 20 queries to mysql I wanted to read the whole table into a simple array ( array index being the record id). There is no write process, just select statements. Or the fetching from the array in its place as I thought would be the fasted way to get it done. But Perl seems to take a lot more ram than the data on disk and I have only 8G.

Any suggestions or hints on how to get the data into memory? Or is that the wrong approach?
Here is what I tried:
my $sql="SELECT * FROM product";

my $sth = $dbh->prepare( $sql ) or die "Couldn't prepare statement: " 
+. $dbh->errstr;

$sth->execute() or die "Could not execute SQL statement:$sql \n" . $st
+h->errstr;

my $rv = $sth->rows;
my $count = 0;
my $rowcache = []; # cache for batches of rows

# while ( my $row = shift(@$rowcache) || shift(@{$rowcache=$sth->fetch
+all_arrayref(undef, 10000) || []}) ){

while ( my @row = $sth->fetchrow_array() ){

#    $content[$count] = \@row;

#    $count++;

    $count = push(@content, \@row);

    print "count = $count\n";

}

[download]
But after about 7M rows I started to run out of memory.... The I tried processing small junks (1000 rows at a time) but the same problem.

I am fairly now to Perl but don't mind doing the reading & work so just some hints would much appreciated !

Thx in advance for any help!

In reply to read a huge mysql Table into memory by hiX0r

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.