This a pretty easy to do using threads, but any sort of demo requires the answers to a few questions:
That is, will the consuming process be reading fixed-sized blocks of data or variable length lines?
If the throttling process runs out of buffered data and so is unable to sustain the required data rate is this fatal to the consuming process or just inconvenient?
If the data source can periodically 'dry up' for extended periods, then several strategies are possible depending upon the nature of the consuming process' tolorance to starvation.
Basically, you delay the output until a substantial quantity of data has been buffered before starting to supply the consumer. Ultimately, you read the whole of the source before supplying anything to the sink at the required rate.
You measure the rate at which data is being received over a shortish timescale and adjust the output rate up or down to ensure that even when the data source dries for an extended period, the sink receives something at regular intervals. Even if the total throughput is reduced to a trickle until the source speeds up again.
Alternatively, the sink may require fixed amounts of data to process, and in the event of the source slowing, it may be better to deliver those fixed size chunks at a slower rate until the source picks up again.
For some processes, a steady rate of data may be more important than the actual data itself, in which case it may be better to repeat data in order to sustain the rate when the source dries.
A similar set of strategies are available for dealing with short periods of over production by the source.
Ultimately, if the source produces faster than the sink consumes, it may be necessary to discard data periodically to prevent memory exhaustion.
A description of the application would clarify many of these questions.
In reply to Re: how to throttle/cache a simple HTTP datastream
by BrowserUk
in thread how to throttle/cache a simple HTTP datastream
by intel
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |