neversaint:
When I reach problems where the data is no longer willing to fit into the available RAM, I normally consider several alternatives:
- Stuffing the data into a database and express the operations in SQL, and letting a DB server do the hard lifting, or
- Examining the problem to see if it is amenable to having the arrays partitioned, operated upon and then recombined, or
- Examining the problem to see if it can be operated on in a stream fashion. (Example: IIRC, array multiplication can be done in a streaming fashion by scanning the array and tracking the partial results for rows & columns, and generating the results from the row & column partial results.)
...
roboticus