Re: [OT] The statistics of hashing.

Is your ultimate goal to simply estimate how many of the strings are unique (or conversely, how many are duplicates), and possibly provide a confidence measure for the estimate? Or are you hoping to come up with an exact number of actual uniques/duplicates for the whole set? (Based on your stated plan, it seems clear that you aren't trying to identify/enumerate the unique strings.) If you're just looking for an estimate, why not work on a manageable and reasonably sampled subset of the data?

In my linguistic background, "several" usually means "more than 4, less than 10". If that's what it means for you as well, and if syphilis has provided a valid metric, then once you're done processing the first 4 billion strings, each new string will definitely be counted as a duplicate, regardless of whether it's a true or false positive. (Perhaps that's not exactly an absolute certainty, but it seems close enough.)

Comment on Re: [OT] The statistics of hashing.

Replies are listed 'Best First'.
Re^2: [OT] The statistics of hashing. by BrowserUk (Patriarch) on Apr 01, 2012 at 08:26 UTC
If you're just looking for an estimate, why not work on a manageable and reasonably sampled subset of the data? Because, according to (my application of) the formulae I found on line, I should be seeing false positive after low 10s of millions of inserts. But my testing showed none after 1e6, 10e6, 100e6, 200e6 ... Ie. My sampling showed that the calculations were wrong, but provided no information upon which to draw any further conclusions. The nature of the beast is that I needed to run the experiment for 3/4 of a billion iterations in order to get my first piece of actual information from it. This start out being about verify my gut with respect to the big strings. That lead to looking for a mechanism capable of verifying uniqueness with the huge numbers involved. That lead -- through various documented mechanisms that proved inadequate -- to coming up with the current schema that appears to work substantially better than all the documented mechanisms. This post is about trying to come up with math that fits the experimental evidence. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]

Replies are listed 'Best First'.

Re^2: [OT] The statistics of hashing.
by BrowserUk (Patriarch) on Apr 01, 2012 at 08:26 UTC

If you're just looking for an estimate, why not work on a manageable and reasonably sampled subset of the data?

Because, according to (my application of) the formulae I found on line, I should be seeing false positive after low 10s of millions of inserts. But my testing showed none after 1e6, 10e6, 100e6, 200e6 ...

Ie. My sampling showed that the calculations were wrong, but provided no information upon which to draw any further conclusions.

The nature of the beast is that I needed to run the experiment for 3/4 of a billion iterations in order to get my first piece of actual information from it.

This start out being about verify my gut with respect to the big strings. That lead to looking for a mechanism capable of verifying uniqueness with the huge numbers involved. That lead -- through various documented mechanisms that proved inadequate -- to coming up with the current schema that appears to work substantially better than all the documented mechanisms.

This post is about trying to come up with math that fits the experimental evidence.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

[reply]