This is basically a choice between storing all the data in memory at once, or running thousands of iterations of a query. Even if you merge the tables into 1 table, you'll still have to sort the results by date, which means storing everything in memory at one point or another (unless items are submitted in date order). About the only thing you can choose is whether to use SQL for the sorting or Perl. SQL is more efficient in terms of memory.

I'd personally use 1 table with fields for date, variable ID, and data point, instead of 5 different tables with fields for date and data point. I'm assuming the data points can all be put in the same field type, since you want to compare them. Then just query for everything sorted by date and process the results.

In reply to Re: Efficient Data Comparison by TedPride
in thread Efficient Data Comparison by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.