This is basically a choice between storing all the data in memory at once, or running thousands of iterations of a query. Even if you merge the tables into 1 table, you'll still have to sort the results by date, which means storing everything in memory at one point or another (unless items are submitted in date order). About the only thing you can choose is whether to use SQL for the sorting or Perl. SQL is more efficient in terms of memory.

I'd personally use 1 table with fields for date, variable ID, and data point, instead of 5 different tables with fields for date and data point. I'm assuming the data points can all be put in the same field type, since you want to compare them. Then just query for everything sorted by date and process the results.