This is no explaination, but it may help. When you concentrate on a sample instead of the entire population, you're doing two estimates: the mean and the variance (square of standard deviation). The issue is that when you estimate the variance, you're subtracting the estimated mean from each item in the sample: you're using an estimation inside another estimation.

Which leads us to the concept of degree of freedom. The sample has N degrees of freedom, i.e. N possibility to be modified: you can have different values for each of the N items. Thus, when you estimate the mean value, you divide by N.

When you estimate the variance, you're using the mean value evaluated over the sample, as said. Given the fact that you're implicitly trusting that mean value to be correct (otherwise you'd not use it to evaluate the variance!), you're stealing a degree of freedom. I mean: if you fix the value of the mean, you can move only (N-1) items, and the N-th will be bound to have a value that leads to the given mean value. Thus, a variance evaluated in this way only takes into account the variations brought by (N-1) items, not N.

Hope that this intuitively helps :)

Flavio
perl -ple'$_=reverse' <<<ti.xittelop@oivalf

Don't fool yourself.

In reply to Re^6: standard deviation accuracy question by polettix
in thread standard deviation accuracy question by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.