The best intuitive explanation I have come across for why dividing by N the sum of squared deviations from the sample mean underestimates the population variance is that the sample mean "follows" the sample; i.e. the sample almost always deviates from its own mean less than it deviates from the population mean (and it never deviates more). This is the source of the bias frodo72 alluded to.

This intuitive argument only shows that simply taking the sample average of squared deviations from the sample mean will underestimate the population variance, but it does not at all prove that N/(N − 1) is the right correction factor. I don't know of an intuitive argument for this, but a nice rigorous derivation can be found here.

the lowliest monk

In reply to Re^6: standard deviation accuracy question by tlm
in thread standard deviation accuracy question by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.