in reply to Re^3: "Automated" benchmarking
in thread "Automated" benchmarking

The overhead introduced by Devel::SmallProf for every Perl sentence executed is too high resulting in a non-linear distortion of time measurements.

Replies are listed 'Best First'.
Re^5: "Automated" benchmarking
by BrowserUk (Patriarch) on Jan 24, 2012 at 13:12 UTC
    a non-linear distortion of time measurements.

    I'd be interested to know more about the "non-linearity" if you have the time to explain further.

    But for the most part, the absolute accuracy of the timings is of little importance to profiling.

    The important factor is the relative timing, and the consistency those relative timings from run to run.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      Say you have two Perl sentences A and B. A runs in 10us and is executed 20 times. B runs in 100us and is executed 10 times. Now suppose that the overhead introduced by Devel::SmallProf is 100us per sentence.

      Then:

      real_time(A) = 20 * 20 = 400us real_time(B) = 100 * 10 = 1000us measured_time(A) = ( 20 + 100) * 20 = 2400us measured_time(B) = (100 + 100) * 10 = 2000us

      So, you should be optimizing B, but Devel::SmallProf tells you to look at A.

      The current implementation of Devel::SmallProf performs a sub call, a call to caller, a hash lookup, a couple of tests and conditionals and two calls to Time::HiRes::time() for every sentence making the timings completely unreliable.

      I tried to reduce that overhead as much as possible on Devel::FastProf writing the DB sub in C/XS. But on Devel::NYTProf, they went further (IIRC) replacing Perl OP loop by their own and eliminating much of the overhead of the DB interface.

        Thank you for the concise and clear explanation. But ... :)

        Isn't the 'nulltime' compensation code meant to correct for (most of) that?

        # "Null time" compensation code $nulltime = 0; for (1..100) { my($u,$s,$cu,$cs) = times; $cstart = $u+$s+$cu+$cs; $start = Time::HiRes::time; &$testDB; ($u,$s,$cu,$cs) = times; $cdone = $u+$s+$cu+$cs; $done = Time::HiRes::time; $diff = $done - $start; $nulltime += $diff; } $nulltime /= 100; ... my($delta); $delta = $done - $start; $delta = ($delta > $nulltime) ? $delta - $nulltime : 0; ...

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

    A reply falls below the community's threshold of quality. You may see it by logging in.