in reply to Calculating Completion of Feeds with Varying Volumes
The only way to know the progress of a feed is to somehow first ask the feed source how much data it has to send. Or, if you control that source, simply include the percent-done information as part of the records in the feed. Either way, the only party that knows the answer, and therefore must provide it, is the source itself. A simple number of bytes received just might be as useful to the human operators as anything; they would quickly learn how much data to expect. Storing the number of bytes received from recent feeds might be useful both as a diagnostic and as a predictor.