in reply to Re: Handling HUGE amounts of data
in thread Handling HUGE amounts of data
If the values in line N of @aod are dependent on the values in line N-1 plus a random factor plus the corresponding line in @aob, then all you ever need to do is keep three lines in memory: (1) the previously constructed line of @aod (2) the line currently being constructed (3) the corresponding line of @aob. Your algorithm would look something like this:
As for unpacking, that depends on what you are doing with the data. However, if you can come up with a way to use the data only one or two lines at a time, then even during the unpack phase you won't need much memory. Just read in the number of bytes per line, unpack, process, and discard.
|
|---|