in reply to Speeding up data lookups

Another thought: if you have a large number of queries working against the exact same data, you could consider indexing the queries instead of the data, then make a single pass over the data to see which records match which queries, and send those records to the appropriate place. This is an idea from streaming databases, but it might work for you.

For example, if your queries are all for ranges of records in your file, you could index which queries start and stop at which records, then inspect each record, see if any queries should start or stop on that record, then send it to all active queries. I'm sure real life won't be so simple, but you get the idea.