Of course the solution strongly depends on your exact problem (how long are the words? how often do they change? how much time can you invest in building an index? how often do you run queries?).

One thing you could do is a sorted file of fixed length records in which you perform a binary search.

You can use a hash to cache the most often used search terms, and a index for the first 2**16 (or whatever) positions in the file that a binary search would visit.

Or you just use a database engine in the hope that it implements the index very efficiently.

(I seem to recall that we had a similar question recently, I'll see if I can find the link). Update: here it is: fast lookups in files. It has some very good answers and is worth a read.

In reply to Re: Best way to look-up a string amongst 100 million of its peers by moritz
in thread Best way to look-up a string amongst 100 million of its peers by anson

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.