Recently we ended up with a matrix of 4 TB (calibration information for calibrating infrared images). In theory it would have fit on the server but doing matrix operations on a matrix of that size is asking for trouble. So we had to retrace are steps, give it another thought and we could reduce the size dramatically. Turned out that only a fraction of the data was really what we needed.
In your case: 2^40 = 1099511627776.
So this would result in about 10^12 hash entries. I assume you need to do lots of searches in the data. (Why else the hash?)
Now I don’t know how much Perl internally uses for representing a hash. But let’s take a small value like 10 bytes for each hash entry (key value pair). (It’s probably more!). Then this already results in approximately 10 TB’s (2^40 / 1000^4). Clearly a Perl hash is not the way forward.
BrowserUK suggested to use the file system. I have implemented something similar in the past. The size was “only” one TB and it worked fine. But there was a difference; the tree structure was very simple in my case, only a few levels deep and the files stored where relatively big (tiff images). Storing many small files in a deeper tree.... I don’t know.
The best approach to me seems the VLDB approach as suggested by swampyankee. Because that’s basically what you have/want: a VERY LARGE DB.
It's probably a good idea to rethink your problem.
In reply to Re: Hash entries starting with a given string.
by dHarry
in thread Hash entries starting with a given string.
by SwiftEagle
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |