EvdB has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I recently ran Benchmark on some of my code and am curious as to how best to look at the results.

In the follwing code $link is an object which can be loaded from the database or the cache using the two private functions benchmarked and $criteria is a hashhref.

use Benchmark qw(cmpthese);

cmpthese( 10_000,
    {
      'From Cache' => sub{ $link->_load_from_database( $criteria ); },
      ' From DBI' => sub{ $link->_load_from_cache ( $criteria ); }
    }
  );

Update:

use Benchmark qw(cmpthese); cmpthese( 10_000, { 'From Cache' => sub{ $link->_load_from_cache ( $criteria ); } +, ' From DBI' => sub{ $link->_load_from_database( $criteria ); } } );
The results of the benchmarking are:
Benchmark: timing 10000 iterations of From DBI, From Cache... From DBI: 16 wallclock secs (13.66 usr + 1.50 sys = 15.16 CPU) @ 65 +9.63/s (n=10000) From Cache: 16 wallclock secs ( 6.86 usr + 0.75 sys = 7.61 CPU) @ 13 +14.06/s (n=10000) Rate From DBI From Cache From DBI 660/s -- -50% From Cache 1314/s 99% --
As you can see the Cache code is much faster at 1314/sec (DBI is 660/sec). However the wall clock times for both are the same, and as 10_000 iterations were timed this would imply that in user time there is not much to choose between them.

My question is "how should I look at these results?". What generally accounts for time that Benchmark does not see, and which time result should I use in making performance questions. I hope that this is enough detail for a general discussion as opposed to specifically looking at the code I posted.

-- tidiness is the memory loss of environmental mnemonics

Update per author - dvergin 2003-02-12

Replies are listed 'Best First'.
Re: Interpreting Benchmark results. (oops)
by tye (Sage) on Feb 12, 2003 at 15:27 UTC
    'From Cache' => sub{ $link->_load_from_database( $criteria ); }, ' From DBI' => sub{ $link->_load_from_cache ( $criteria ); }

    Just wanted to note that it looks to me like your results are reversed. Oops.

                    - tye
      You are absolutely right. I spotted it just after posting and have a 'fix it' lodged with the appropriate wing of this vast institution.

      -- tidiness is the memory loss of environmental mnemonics

Re: Interpreting Benchmark results.
by l2kashe (Deacon) on Feb 12, 2003 at 16:23 UTC
    Another really really nice module I found while looking for datadumper was Devel::AutoProfiler. It basicly examines and profiles sub routine calls.

    The use of it is simple, your first line after interpreter intialization is use Devel::AutoProfiler. It will then profile and time all calls your code makes during execution. If you have say 'use strict' and you dont want to profile the strict module, simply place the use statement after the use strict (examples below).

    I recently started in on creating a MUD in perl, just cause it's a decent excercise. With the use of Devel::AutoProfiler, I have already sped up my program in a few different places. It isn't a really fine grained tool, but it can be very usefull to get an overall feel for whats going on within your program at a high level, and provide points to attack within your codebase.

    Note: requires Time::HiRes
    #!/usr/bin/perl use Devel::AutoProfiler; &do_it(); sub do_it { print "Hello world\n"; for ( 1 .. 20 ) { print "$_\n"; } } # OUTPUT BELOW : I have removed code output for legibility $VAR1 = [ 'main::do_it', { 'total_calls' => '1', 'file -, line 3' => '1', 'total_time_in_sub' => '0.003332' } ];
    Example 2
    #!/usr/bin/perl # We dont want to profile strict, so place the autoprofiler # line after that module. use strict; use Devel::AutoProfiler; for ( 1 .. 20 ) { &read_file('/etc/passwd'); } sub read_file { my $file = shift(); open(IN, "$file") || die "Cant access $file\nReason: $!\n"; while (<IN>) { print "$_"; } close(IN); return; } # Added comments to profiler output and apologies if people # dont like the overuse of # signs # OUTPUT BELOW again without actuall code output # $VAR1 = [ # # The subroutine name called # 'main::read_file', { # # file and line in codebase that routine was called from, # I did this via STDIN, so file is a -, and the call was on # line 6 # 'file -, line 6' => '20', # # number of times the routine was called # 'total_calls' => '20', # # the total time it spent in this particular sub #(in seconds) # 'total_time_in_sub' => '0.090052' } ];
    Granted these are contrived examples off the top of my head, but if you were to have a few sub routines nested, and one is taking an inordinate amount of time to run, at least you can see it.

    Benchmark is a great tool, but I sometimes find it hard to take large/complex segments of code and pass it off the same way it would in my program. With AutoProfiler, I simply run the code and can quickly see where my time is taken up.

    Hope this helps some, even though it isn't directly related to reading the output from Benchmark.

    /* And the Creator, against his better judgement, wrote man.c */
Re: Interpreting Benchmark results.
by jasonk (Parson) on Feb 12, 2003 at 15:04 UTC
    The DBI test spent 15.16 seconds of it's 16 seconds actually running, the cache one only spent 7.61 seconds on the CPU, and probably spent the rest waiting for something (disk access perhaps). At the rate your tests run, I would probably benchmark a lot more than 10k iterations, you would probably get better results.
Re: Interpreting Benchmark results.
by hossman (Prior) on Feb 13, 2003 at 02:12 UTC
    In no particular order...
    • Wallclocks are the elapsed time from start to end. Which is usually what you want if you are trying to profile the difference between reading data from a file vs a DB, because you care about the acctual time spent. But you have to be warry of wallclock seconds, because if your system is doing nything else while you run the test (ie: if you start surfing the net while waiting for the test to finish) then it will skew the results.
    • It's very concievable that your DB time might be faster then your "cache" time -- if you cache is on disk, and your DB caches things in memory.
    • Someone else in this thread (i don't remember who at the moment) mentioned that there are ways of profilifng your whole program, not just snippets -- this is a very good idea, for two reasons:
      1. You might spend a lot of time optimizing what you think is the slow part of your code, only to find out later that something else is where the real time is wasted.
      2. running N iterations of hte same piece of code may not give you an accurate picture of what happens when your programm runs "in the wild". if you are working on a DB based App, and you have a caching mechanism for dealing with followup lookups on the same data, then running db_lookup(42) 10,000 times followed by the cache_lookup(42) 10,000 times won't neccessary give you a good picture of reality -- the hardware might start caching the disk page the file is on, or the DB might start caching the rows in memory, making your times much faster then they normally would be if you did 10,000 calls to each on 10,000 different Ids.
    Bottom line: make sure what you are profiling makes sense, and is what you want/need to be profiling.
Re: Interpreting Benchmark results.
by !unlike (Beadle) on Feb 12, 2003 at 15:34 UTC

    I don't use Benchmark that much (not really an issue for the code I write) so I'm not familar with the finer aspects of the module.

    Personally if I would look to the table produced at the end of the output over the wallclock secs

    As jasonk suggested, running through more iterations may give a clearer difference in results

    !unlike

    "The price if ignorance, which is of course that you must learn from those who know." Socrates (paraphrased)