in reply to Calculating the average on a timeslice of data
All things considered, you might wish to take a long, hard look at SQLite. This is, believe it or not, a public domain database system which stores everything in a single file and does not involve a server. So, you get the goodness of “a flat file,” and the power of SQL. You also have very nice interactive utilities available. I find myself using it almost exclusively these days, in situations where I would normally use a flat file, and I almost always find the project taking a positive shift into a new direction because I can push more of the hard work off to SQL queries.
The only thing to bear in mind when using it ... but it is a big thing ... is the importance of “transactions.” If you are updating or inserting something, you need to be doing it within a transaction.