It depends on the thing you are searching - that is how are you searching it. I've tried DBM::Deep for prefix search exactly because of the speed it's docs mention. However, when tested, even from a RAM disk (aka /dev/shm) it was at least two/tree orders of magnitude (not sure exactly) slower than loading up everything in HoHoH... http://en.wikipedia.org/wiki/Trie and searching by going "up the tree".

Of course, I had the luxury or all requests going through one process that's doing prefix search and routing requests depending on it. Which might also be a solution for the original question.

You could implement the mod_perl handler only to pass on the requests, well put in work queue, and wait for results in results queue. And have one process (or several, possibly on several different servers) that do the actual (is it prefix?) searches - reading requests from the in/work queue, and putting them back in the out_queue.

The trie implementation was able to find something like 50K random prefixes per second in a loop - from the pool of 45 to 50K prefixes in the database (loaded on startup into large HoH trie). 50.000 per second should outperform any web server. And that's on x2 AMD with 1 GB or RAM ...

You could use Memcache for both input queue, and output - implementation (in Perl!) can be seen here: http://3.rdrail.net/blog/memcached-based-message-queues/

PS - I've been contracting for the past year, for an company that's in SMS gateway business. So I get to tweak code that searches prefixes a lot ;)

Have you tried freelancing/outsourcing? Check out Scriptlance - I work there since 2003. For more info about Scriptlance and freelancing in general check out my home node.

In reply to Re^2: large perl module by techcode
in thread large perl module by minek

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.