I wasn't sure myself, so I just did a simple-minded test, and sure enough, when the child starts up, it takes up as much memory as the parent, which means that you're getting a full copy of your 2gb in-memory data each time you fork. Forking 5 children would pretty much guarantee that the OS will need to do a lot of memory swapping to run all those huge child processes. I think the delays you're seeing are not so much the cpu load of the children, but rather the i/o wait imposed by swapping. (Some versions of "top" will report the total percentage of processing time devoted to "i/o wait" -- if your version of "top" shows that, you'll probably see it skyrocket).

If you want some sort of approach that actually shares a single copy of the 2GB data set among multiple clients that are being served simultaneously, I think you'll need threads rather than forking. I'm not a reliable source on this, 'cuz I've never used threads myself, but... if I'm not mistaken (no guarantee on that), one of the advantages of threading is that you really can share a single store of in-memory data across threads, whereas you can't do that across children forked from a given parent. I hope others can elaborate from personal experience...

Meanwhile, you may want to reassess your requirements. How important is it, really, for multiple clients to be serviced in parallel (given that doing so might not be doable without a serious loss of efficiency)? Is there any chance the process could work from a mysql database, rather than from in-memory storage? (Multiple concurrent access to a 2gb dataset is a lot easier to implement efficiently using a real RDBMS, and mysql is pretty zippy for a lot of tasks.)

In reply to Re: Handling multiple clients by graff
in thread Handling multiple clients by jalewis2

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.