Recently I was trying to profile a script and noticed that when I ran the script two or more times in sequence the total time and the breakdown between the different start-up (perl) and actual execution functions seems to change with each run. On a short script the variance can be as much as 30-40% (e.g. ranging between 0.040s-0.090s). Even on longer running scripts the variance is often in the 10% range.

This variance happens even on a single user machine with nothing but OS related background processes running. Obviously, those can't just go away, so I presume this variance is just a given of the profiling process.

However, this raises a question for me. How do I tell if a code change really improves performance? If I take only one profiling result before and after I can't really tell if the "improvement" is due to the code change or just an artifact of the background processes and resource sharing at the time of each profiling run.

Of course, I could do several profiling runs before and after and take averages. Is this something others do? Is there software designed to run profiling several times and calculate the statistics? And if we are going the statistics route, how many runs are needed to get a reliable result? Is an average really the best "central tendency" for comparing before and after results (alternatives: median, mode, min, max)?

Many thanks in advance, beth

In reply to What is the best way to compare profiling results before and after a code change? by ELISHEVA

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.