Thank you all for your replies.

GrandFather's solution is interesting and I will give it a try. Definitely did not think of doing a check that way.

The reason I have created this script is to track feeds which are trending in one direction or another at some varying rate - it is difficult to set a constant value to check against. Hence, I have created a neural network for each feed which uses extensive historical data to make fairly accurate predictions of feed volumes for the following day.

To answer your questions ww:

The significance of marking a feed "completed" is that I can stop worrying about it not being finished on time. I have a similar NN predicting a recv'd time for each feed as well so that I know when I should start worrying about a feed being overly late and can contact the content provider. There is no automated action that I would risk linking to this script, it is simply feeding a webapp that I use for monitoring.

That said, I am willing to risk the possibility of a mistakenly marking a feed "complete." This is very rare due to a feature of the feeds - the majority of the files will come in 1 or 2 of the updates. It is usually the case that after I recv these updates that the feed can be marked complete.

Waiting for actual completion is generally not an option due to the feature described above. The final files of the feed typically arrive much, much later than the bulk and I am only concerned with having recv'd that majority of files as it generally guarantees that the rest will follow. So it wouldn't be a small increment to wait =)

Also, I do not control these feeds, nor do I have a way to contact the owners for a total size.

So I was thinking that there must be a way to create some sort of continuous sliding tolerance value which I could use to calculate acceptable "complete" volumes for feeds of all sizes.

In reply to Re: Calculating Completion of Feeds with Varying Volumes by temporal
in thread Calculating Completion of Feeds with Varying Volumes by temporal

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.