in reply to Re: Hash Clash on purpose
in thread Hash Clash on purpose

First of all I am too lazy to actually test whether changing the initial value changes the relative choices of hash buckets. I strongly suspect it does, but I either have to work it out by hand or else install a bunch of stuff on this computer (eg Perl) and then write a program. What we really want to do is tell that it sends what had been 0 buckets back to 0, and for that I should find those strings which means installing stuff, which I haven't done...

However assuming that the analysis shows that changing the initialization value does change hashing decisions, your pseudo-code looks wrong to me. You are initializing it randomly per hash lookup. For hashing to work, the hash lookup algorithm has to be consistent from lookup to lookup. Instead what you need to do is save the value of the initial value somewhere and then pull that into hash_PeRlHaSh.

That means that you have to store that somewhere. Several options exist. One is to reserve space per hash to store its initialization value, and then look that up per lookup. Another is to have a global value chosen for all of your hashes. And a third is to make it a random compile-time constant. Problems with binary linking of XS modules that have been copied from point A to B make the last one infeasible. The first one adds 4 bytes to every hash, which isn't that much, but we have a lot of small hashes. An offhand guess is that we would see 1-5% space usage increase, and (of course) binary incompatibility.

The middle option (a run-time random constant) looks to be the best bet. p5p might have some arguments over binary compatibility (code compiled with hash lookups initialized to 0 won't match code compiled with your new initialization) but it should be easy to have whether to initialize randomly or to 0 at startup to be a compile-time flag.

Hmmm...looks like I argued myself into believing that you can fix the problem with your approach. It would be worthwhile for you to try to make the fix, run tests to verify that it does make the performance attack infeasible, then try to get it accepted... :-)

Replies are listed 'Best First'.
Re: Re: Re: Hash Clash on purpose
by BrowserUk (Patriarch) on Jun 03, 2003 at 17:58 UTC

    Agreed. My pseudo-code wasn't up to much.

    However, I just remembered/re-discovered an interesting thing. In common with CV's, SV's and AV's, the HV structure has an NVX field. This is the 32-bit field used by SV's to store the numeric value of SV's once they have used in a numeric context. In the HV, it is unused.

    It seems to me that this would be the perfect place to store a (hash) creation time generated random value that could be used to initialise the hash function.

    That would effectively allow for a different hashing function for every hash within a program, and for every run, with no extra space required, and only hits would be:

    A register indirected load of the initialisation value at every hash generation, which shouldn't more than a few clock cycles on any processor as it should already be in the level 1 cache as soon as the HV has been addressed.

    The one-time cost of generating the random value when the hash is created.

    The only real argument against the idea that I forsee is that it might be possible that an application could randomly hit worse case performance. Statistically, the odds of this happening are, I think much the same as now for any given dataset, but the possibility that an application that loaded that same hash with the same data every run, could randomly have that hash degenerate to worst case, could be seen as a problem.

    I really wish I was set up to be able to build, test and propose this fix to p5p, but I'm not, and am unlikely to be in the near future. Maybe Elian or somebody can pass the idea along if they think it has some merit.

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller


[reply]

In Section Meditations