benchmark ldap search

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I want to time the length of a series of ldap searches. This is my first time using benchmark and I wanted to make sure this was the best way of doing it. Do I have to time each search more than once to get accurate reads? Here is a snipet of the code

$ldap = Net::LDAPS->new(HOST,port => PORT,timeout => TIMEOUT) or die "
+Can't connect to LDAP server " . HOST;
$ldap->bind(DN, password => PASSWORD, version => 3);
open(S, "subnet.log");
while (<S>){
        chomp $_;
        my $results = 
 timeit(
       1, q{my $base = $_;
        my $look= $ldap->search(base => $base,
        filter =>('objectClass=dNIPIPAddressConfiguration'),
        attrs => ["cn"])}
        );

print "$_:" . timestr($results) . "\n";
}
[download]

Comment on benchmark ldap search Download Code

Replies are listed 'Best First'.
Re: benchmark ldap search by Old_Gray_Bear (Bishop) on Nov 03, 2003 at 16:32 UTC
You really will need to run multiple times if you are looking for accurate timings. There are a few things that will effect the repeatability of your result - Network load (between you and your LDAP server) load due to back-ground tasks on your client machine and the server. I don't know of any reliable way to remove these effects, so you will have to make multiple runs and statistically analyze the results to get a good set of numbers (average time for the search,standard deviation, and a correlation coefficient or chi-squared coefficient). Also bear in mind that this result is true (in some sense of Truth) only for the particular search filter you used. Searching for a different set of attributes will give you different numbers. If you are trying to benchmark different LDAP servers, you will want to run the benchmark multiple times with several different search-filters against different attributes types to get a better picture of the respective abilities. The last time I was involved in this process, we were evaluating three different vendors. I built four different search-filters (an approximation of the kinds of things that the Users searched for routinely). We ran each filter ten times against each product, with the client and server each on a dedicated machine, but using the corporate back-bone to connect them. That gave us 3 x 4 x 10 runs to evaluate. If I were to do it again, I would look into adding at least one more filter and argue for at least 15 more runs per filter. That would improve my correlation statistic to 0.90 ( from 0.875). They are not numbers that I am particularly happy with, but we only had one weekend to run the evaluations for ourselves. The statistics did give use a feel for the performance of the three products, and Management needed something to justify the decision upon. I Go Back to Sleep, Now. OGB	[reply]

Replies are listed 'Best First'.

Re: benchmark ldap search
by Old_Gray_Bear (Bishop) on Nov 03, 2003 at 16:32 UTC

I don't know of any reliable way to remove these effects, so you will have to make multiple runs and statistically analyze the results to get a good set of numbers (average time for the search,standard deviation, and a correlation coefficient or chi-squared coefficient).

Also bear in mind that this result is true (in some sense of Truth) only for the particular search filter you used. Searching for a different set of attributes will give you different numbers.

If you are trying to benchmark different LDAP servers, you will want to run the benchmark multiple times with several different search-filters against different attributes types to get a better picture of the respective abilities.

The last time I was involved in this process, we were evaluating three different vendors. I built four different search-filters (an approximation of the kinds of things that the Users searched for routinely). We ran each filter ten times against each product, with the client and server each on a dedicated machine, but using the corporate back-bone to connect them. That gave us 3 x 4 x 10 runs to evaluate. If I were to do it again, I would look into adding at least one more filter and argue for at least 15 more runs per filter. That would improve my correlation statistic to 0.90 ( from 0.875).

They are not numbers that I am particularly happy with, but we only had one weekend to run the evaluations for ourselves. The statistics did give use a feel for the performance of the three products, and Management needed something to justify the decision upon.

I Go Back to Sleep, Now.

OGB

[reply]