How would you write a test that checks whether the cache is actually used? Clearly you cannot access %cache to test as its lexical in the sub.

Scanning the other postings, it appears that part of the requirement is that the code cannot be changed to instrument or otherwise report on its behaviour.

Also, you've got other tests to check that the returned values are correct.

So the purpose of this testing is simply to check that:

  return $cache{$arg} if exists $cache{$arg};

[download]

and:

  $cache{$arg} = $result;

[download]

do what they're expected to do.

So why wouldn't they ? Is it possible for $arg to be something that doesn't map well to a hash key... an object, for example ? Can take values that are equivalent, but don't look the same when stringified ? Is it possible for $result to be something that cannot be stored in a hash, or whose validity or integrity is not guaranteed over time ?

The code as given doesn't betray any doubt that all arguments and results are suitable.

So, I guess the testing you have in mind is essentially:

  1. Try foo(bar), where we know this is the first time bar has been evaluated. Check (a) that result did not come from the cache, and (b) that the result ends up in the cache.
  2. Try foo(rab), as above.
  3. Try foo(rab) again. Check (a) that the result came from the cache, and (b) that it's the expected value.
  4. Try foo(bar) again, also as above.

Unless we're concerned that the cache mechanism is not deterministic, that seems enough to me. This quick sequence of tests checks that: rubbish from the cache is not returned the first time a given argument is presented; that it isn't just returning either the first or the last result it put in the cache; and that the cache holds more than one result. For the amusement value, however, this could be expanded to a larger number of randomly selected arguments, presented in a random(ish) order.

In fact, it doesn't seem necessary to check that the result is recorded the first time foo() is called for a given argument; if it wasn't, a later test would fail because the result would be recalculated. (This is just as well, as the cache is hidden inside the subroutine, and you've rejected the Padwalker as a way to poke at it.)

So, the one piece of information that each test needs is: was the result established anew, or did it come from the cache ? If the operation something costly is reliably many, many cycles then user CPU time appears to be the one thing that a completely black-box test can use.

(I am assuming here that the testing can be run from a known state of the cache -- empty being the obvious one.)

Timing doesn't seem wholely satisfactory. So, what I would do is introduce a counter (that can be read by the test) to tick each time a result has to be established. Making this a permanent part of the code means that the test is entirely external. Incrementing a counter each time the something costly executes seems unlikely to be a big overhead.

Actually, I'd be tempted to put in a second counter, either to count total number of calls of foo() or to count the cache hits -- which could be interesting for ongoing monitoring of the performance of the program.

Mind you, having stepped through the code with the debugger, and established that both cache-hit and cache-miss cases do what they are supposed to do, I'd wonder what it was needed testing... It would be different if the code were more complicated... for example: if there was code to age out cache entries, which I'd be tempted to stress test.

In reply to Re: How to test caching? by gone2015
in thread How to test caching? by JavaFan

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.