I've used locality-sensitive hashing before; that might be something like what you remember. As for what you actually want... if I knew how to do it, I'd probably be charging a lot of money for it.
Comment on Re^3: What is the best way to store and look-up a "similarity vector"?