in reply to Hash of Arrays - Pushing Array references and memory impacts question

There is a bit of a catch-22 situation here.   The real problem with “memory” is that in the end it is virtual memory, with randomly-occurring page faults depending on just how much “locality of reference” your program displays (as well as the total amount of the memory resource you are consuming vs. the amount available in the system).   Even when you “tie” an array/hash, the access pattern is still random, so there is still a lot of in-memory buffering going on, thus, still a lot of paging activity and a large working-set size.

I think that you will just have to re-design the thing... take a fundamentally different approach.   You might need to work with “80,000 data sets,” but surely they don’t have to be “in memory” simultaneously.   If you are comparing two data sets, surely there is opportunity for a (SQLite...?) JOIN, and so forth.   Avoid doing “sorting and searching” in-memory, and avoid writing procedural code insofar as possible.