Re^4: Initializing iterations while benchmarking

From a functional perspective this produces the correct result but from a performance perspective it may not. If in the actual code there is only one pass over the array then setting pos is not required. In such a case it is only required for the benchmark. As setting pos takes time that will not be required in the real code, it introduces an error in the benchmark results.

An interesting case is if I want to compare the times with and without pos being set, yet always with the same initial conditions, to learn how much impact setting pos has. This can't be determined without isolating the iterations from each other. Then we are back to the possibility that time to initialize obscures the differences in the code under test.

My apologies for being so fussy - it is not that I don't appreciate your suggestions. I am trying to measure the performance of particular code in isolation and with strictly controlled initial conditions.

Comment on Re^4: Initializing iterations while benchmarking

Replies are listed 'Best First'.
Re^5: Initializing iterations while benchmarking by alexm (Chaplain) on Aug 07, 2009 at 11:05 UTC
As setting pos takes time that will not be required in the real code, it introduces an error in the benchmark results. But pos is actually needed in real code in this case, as proven in strange behavior of grep with global match [resolved], isn't it? Then we are back to the possibility that time to initialize obscures the differences in the code under test. It seems that Benchmark isn't fit for the job, you may need a profiler to mark where you want to start and end mesuring. I have little experience in profilers, so I can't suggest anything at the moment.	[reply]
Re^6: Initializing iterations while benchmarking by ig (Vicar) on Aug 07, 2009 at 11:19 UTC
In the case I was looking at, each string would be tested only once, so no need to set pos before the test. It only became an issue when the same test was performed on the same string as the benchmark test iterated. The broader issue of excluding setup time from the benchmark results is what interests me here. This example is merely what I was looking at when I began to consider the issue. It is a concrete example but otherwise not important in itself. Thanks for sharing your thoughts. update: revised/clarified (I hope) the first paragraph.	[reply]