You have a high performance system. You are blowing data in from the stock market and sucking data out with some analysis program.

In between those two processes is a buffer. I would certainly consider making that buffer of a fixed size. How big it needs to be is determined by the burst rates of in and output.

You are completely lost if the consuming process cannot on average, within a short time window, consume more than the data producing process - the system just cannot work in the high performance mode that you envision - it will "get behind" and you will have buffer overrun problems.

If the buffer starts to panic and ask for even more system resources as it gets too full, this typically will auger into the ground very quickly and there is a system crash. How to loose data as gracefully as possible is one of the considerations. Sorry that I can't help you more. High performance system design is tough - real-time system design is even tougher.

In reply to Re^3: array overhead by Marshall
in thread array overhead by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.