Thank you for taking the time to answer my question. You make an excellent point about variance, but shouldn't the null hypothesis be that there is no (statistically significant) change in mean/median/mode? I'm not sure what you mean by "the variance within each run".

Your idea of comparing the size of variance before and after variance suggests a side effect that I hadn't thought of: changing the code can change the way the script competes with the operating system and may change the variance before and after. I can see that in certain real time situations where timing really matters, you might want to profile that as well as average performance time. However, in my case "usual" performance time, rather than consistency of performance time, is the primary concern.

As for using an ANOVA - that would only apply if the distribution of profiling results is normal. If the distribution is skewed or has overly thick or thin tails, then you would have to use other techniques to analyze the variance. Without knowing the distribution it is very hard to tell how many standard deviations (sqrt of variance) are needed to make the difference between the old and new mean statistically significant (5-7 are needed for a normal distribution).

Best, beth

In reply to Re^2: What is the best way to compare profiling results before and after a code change? by ELISHEVA
in thread What is the best way to compare profiling results before and after a code change? by ELISHEVA

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.