My unsolved quest for efficient variable(memory) sharing on linux continues.

I have a monster (lots of gigs, upto 200gb in future) in-memory hash and a nice perl sub which does lookups and returns results for me.

I want *other* processes to be able to do lookups as well, obviously without every process loading it's own duplicate copy of the hash.

The rate of lookups is extreme, *many* millions per minute at least.

Can anyone think of an efficient way to code this? The "lookup" child processes will be mod_perl (web cgi); I'm hoping the giant-hash process can simply be a local daemon.

I'm wanting to have zero unnecessary overhead: so: no interprocess communication if possible (no sockets, no messages), no memory-shuffling (I don't want to move gigs of hash out of a shared memory pool into the web server pool prior to starting the lookup), no serializing...

Thoughts?

Has anyone ever built a modern perl from source? What about the idea of modding the source to introduce some kind of shared region that way? e.g. every variable with the name "shared_*" uses a different subset of RAM or something like that?

In reply to Highly efficient variable-sharing among processes by cnd

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.