in reply to Re^3: Best way to look-up a string amongst 100 million of its peers
in thread Best way to look-up a string amongst 100 million of its peers

Pre-bucketing and splitting the data can have several advantages over an all-in-one-btree file. Notably, inserts can be faster because there is no rebalancing to be done. And if you split far enough to make the resultant files small, then there is no need to sort the individual files as a linear search is fast enough.

In effect, using the first n chars to index into the appropriate bucket and then linear searching is exactly equivalent to hashing. Ie. You described the same process perl uses for its internal hashes and that CDB uses. Except the hashing part of the process is faster.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."
  • Comment on Re^4: Best way to look-up a string amongst 100 million of its peers