in reply to Calculating Completion of Feeds with Varying Volumes

The only way to know the progress of a feed is to somehow first ask the feed source how much data it has to send.   Or, if you control that source, simply include the percent-done information as part of the records in the feed.   Either way, the only party that knows the answer, and therefore must provide it, is the source itself.   A simple number of bytes received just might be as useful to the human operators as anything; they would quickly learn how much data to expect.   Storing the number of bytes received from recent feeds might be useful both as a diagnostic and as a predictor.
  • Comment on Re: Calculating Completion of Feeds with Varying Volumes