You really will need to run multiple times if you are looking for *accurate* timings. There are a few things that will effect the repeatability of your result - Network load (between you and your LDAP server) load due to back-ground tasks on your client machine and the server.

I don't know of any reliable way to remove these effects, so you will have to make multiple runs and statistically analyze the results to get a good set of numbers (average time for the search,standard deviation, and a correlation coefficient or chi-squared coefficient).

Also bear in mind that this result is true (in some sense of Truth) only for the particular search filter you used. Searching for a different set of attributes will give you different numbers.

If you are trying to benchmark different LDAP servers, you will want to run the benchmark multiple times with several different search-filters against different attributes types to get a better picture of the respective abilities.

The last time I was involved in this process, we were evaluating three different vendors. I built four different search-filters (an approximation of the kinds of things that the Users searched for routinely). We ran each filter ten times against each product, with the client and server each on a dedicated machine, but using the corporate back-bone to connect them. That gave us 3 x 4 x 10 runs to evaluate. If I were to do it again, I would look into adding at least one more filter and argue for at least 15 more runs per filter. That would improve my correlation statistic to 0.90 ( from 0.875).

They are not numbers that I am particularly happy with, but we only had one weekend to run the evaluations for ourselves. The statistics did give use a feel for the performance of the three products, and Management needed something to justify the decision upon.

I Go Back to Sleep, Now.

OGB

In reply to Re: benchmark ldap search by Old_Gray_Bear
in thread benchmark ldap search by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.