This a pretty easy to do using threads, but any sort of demo requires the answers to a few questions:

  1. What data rates are involved from the source?
    • Peak data rate?
    • Required average data rate?
  2. What order of total volumes are involved?
  3. Is the datastream binary or textual?

    That is, will the consuming process be reading fixed-sized blocks of data or variable length lines?

  4. What should be done in the event of underruns?

    If the throttling process runs out of buffered data and so is unable to sustain the required data rate is this fatal to the consuming process or just inconvenient?

    If the data source can periodically 'dry up' for extended periods, then several strategies are possible depending upon the nature of the consuming process' tolorance to starvation.

    1. A large intermediate buffer.

      Basically, you delay the output until a substantial quantity of data has been buffered before starting to supply the consumer. Ultimately, you read the whole of the source before supplying anything to the sink at the required rate.

    2. A feedback loop.

      You measure the rate at which data is being received over a shortish timescale and adjust the output rate up or down to ensure that even when the data source dries for an extended period, the sink receives something at regular intervals. Even if the total throughput is reduced to a trickle until the source speeds up again.

      Alternatively, the sink may require fixed amounts of data to process, and in the event of the source slowing, it may be better to deliver those fixed size chunks at a slower rate until the source picks up again.

    3. Duplicate 'fill' data.

      For some processes, a steady rate of data may be more important than the actual data itself, in which case it may be better to repeat data in order to sustain the rate when the source dries.

  5. What should be done in the event of overruns?

    A similar set of strategies are available for dealing with short periods of over production by the source.

    1. Extend the buffer.
    2. Increase the rate of output.
    3. Increase the volume of output.

    Ultimately, if the source produces faster than the sink consumes, it may be necessary to discard data periodically to prevent memory exhaustion.

A description of the application would clarify many of these questions.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

In reply to Re: how to throttle/cache a simple HTTP datastream by BrowserUk
in thread how to throttle/cache a simple HTTP datastream by intel

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.