|The stupid question is the question not asked|
Actually you may have a point regarding the key. However im not so sure about the ondisk versus in ram issue. Im on a box with large ram, and virtually all of the complex data structures will be reused several times, so my thinking is that I will just let the OS decide if the data should be swapped out or not. Only one of the large strucutures is used at a time, and is used for a reasonable amount of time, so I suspect that avoiding swapping isnt a win at all, and possibly even would be a loss. BTW, the 8000x2 data elements end up being about 100k+ of RAM, but even there its more the construction time than the memory that im worried about. It takes a while to build that 100k structure. (Ive actually contemplated rewriting Tie::Array::PackedC in C, or recoding not to use it at all to reduce the run time, but at this point im overall satisfied with the performance.)
Anyway, more brainfood to digest. Thanks.
First they ignore you, then they laugh at you, then they fight you, then you win.
In reply to Re^2: Bitten by the worst case (or why it pays to know whats inside the black box)