Hi. I've got an optimization problem to solve, and I'm trying to decide on the best approach to solving it. Probably, someone on PerlMonks already knows the answer.

We've got an existing system which loads data by doing individual SQL queries for each record read, and caching results. We have multiple processors on our HP/UX system, so we take advantage of this by running multiple copies of our program, but the results are still far too slow.

I'd like to replace the current implementation with a system that keeps all the configuration data in memory: similar to the database server, but more efficient, and simpler.

To avoid wasting RAM, I'd like to keep a single copy of all the configuration data in RAM, and let every process that needs to read from it.

Three approaches suggest themselves to me: some sort of "ramdisk" type approach, SysV style shared-memory, and memory-mapped files. I haven't worked with any of these on HP/UX, so I'm unsure of their relative merits. I know that SysV shared memory and memory-mapped files are available, but don't know if a ramdisk is a possibility. I do remember that ramdisks under Linux are (or were?) a kernel option that the system administrator needed to configure. I don't administrate the target system, and I don't think re-building a kernel configuration for our production system would be welcomed for our project. So a ramdisk may not be an option, unless someone knows how to configure one under HP/UX without a kernel re-build.

As for mmap and shared memory options, both seem to have some overhead that I don't really need. I specifically don't need to do any tricky inter-process communication: I just want a common section of memory that can be read quickly.

Shared memory under perl seems to copy the data: is this true under C? Specifically, can I just write an XS module to search quickly through the shared memory section, extract the values I want, and return them to perl? Is there a CPAN module that I've overlooked which does this sort of thing?

Memory mapping (via the mmap() system call)seems tied to a file, which may or may not be what I want. It seems to require an open() call and a mmap() call for each filehandle: does this imply that only processes that can share filehandles can share memory in this way? Or will open/mmap detect that the file has been mmapped already?

I don't ever need to write changes back to the file: will loading the file as read-only optimize for this, or will I pay the overhead of the unused write functionality anyway?Is it even significant?

Any suggestions or specific details on how shared memory and/or mmap() work would be appreciated.
--
Ytrew Q. Uiop

In reply to How best to optimize shared data by Ytrew

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.