I will be making a lot of accesses into the file (~5 million?) I know that some lines will be accessed more often than others, but as far as I can tell now, the accesses will be be at least initially uniform. Lines will be looked up more than once, so some smart caching would be beneficial, but until I start making the calculations, Its hard to tell which lines will be accessed more. Of course I have a few big disks, I can just make a few copies of the file and parallelize the task.