Thanks for the feedback and suggestions. It's great to get a variety of responses, as it enables me to test my assumptions against the experience of others. I will be digesting your suggestions over the next week or so.

For those who asked why the data is all read up front, the answer is that it is confidential / sensitive, and loaded from an encrypted file on disk. The decryption happens all in a single call and the decrypted results get stored in an array. I guess we could have done that bit as a filter...

Thanks again,

Jeff