I'll guess that the progressive slowdown looks like the sth_select_hydro SQL, which has to do progressively more work to find the 100 rows that you want.

I would select all the rows first and put the data into a text file, or RAM if you have enough of it.

You mentioned the speed in Matlab being faster. The Matlab code may be smart enough to take advantage of the massive redundancy in your calculation. As you step through the array, your calculation has to operate on what is mostly the same list of numbers over and over. The only points that change are the first and last points in the array. So the clever Matlab routine detects this and does a much smaller calculation, in effect subtracting off the last number from the sum and adding the first number to the sum. It is storing the points for the average in a circular buffer and avoiding the work of recalculation. There are special forms of the statistical formulas for average and standard deviation that enable a result to be incrementally updated. The formulas are in the Wikipedia article on Standard Deviation, in the section 'Rapid Calculation Methods.'

Also, the moving average is going to move very slowly from point to point. Usually, you don't need to know that many points in a moving average, and you don't really need to calculate them all. This is called decimation.

It should work perfectly the first time! - toma

In reply to Re: std dev calculations slow over time by toma
in thread std dev calculations slow over time by punkish

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.