Re: Best way to look-up a string amongst 100 million of its peers

Not enough information.

What range of characters are they constructed from?
What are you looking up?
Just the existance in your list or is there some associated piece of data?
How will the lookups be organised?
Will they be completely random? Or will they come in related bunches?
What kind of app is doing the looking?
A single instance of a long running application doing thoudands of lookups per run.
Many concurrent instances doing one or a dozen lookups per invocation; eg. a CGI?
How fast are you looking to achieve?
The obvious answer is "as fast as possible", but for example, in a cgi app, there are inherent network delays of up to a few seconds, so if (the one or dozen) lookups per run takes milliseconds rather microseconds, your users will not notice.
On the other hand, if the user has to wait for 50,000 lookups, the difference would be significant.
What format is the data currently in?
Can that format change?
Is the list static, or nearly static, or changing hourly, daily, weekly?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: Best way to look-up a string amongst 100 million of its peers

Replies are listed 'Best First'.
Re^2: Best way to look-up a string amongst 100 million of its peers by anson (Initiate) on Mar 26, 2008 at 16:38 UTC
Thanks for all the advice. I had originally tried MySQL and loading the list was fast enough (load data infile...) but I gave up on the indexing process after 24 hours. On your recommendation I have managed to get the list into CDB and the performance of that seems pretty decent! I also love that I don't need to be running a database for such a simple look-up. To answer all your questions about the nature of the data... just go to the URL where I just now got this up and running ;) https://domize.com	[reply]
Re^3: Best way to look-up a string amongst 100 million of its peers by assemble (Friar) on Mar 26, 2008 at 17:40 UTC
I've got to say, that's a pretty nifty little tool :) One suggestion is to make it only lookup after a change if the field hasn't changed again in the last half second or so. I tried a few domains. Some results weren't exactly right, but it's a cool little script nonetheless.	[reply]