Re^2: Testing objects that cache

The first thing you probably want to test is that the cached object and the non-cached object in fact do have the same "black box" behavior.

No, not really. Once the OP is that far, his problem is solved.

His problem is "how do I test whether caching was used"? If you don't know an object you have is a cached object or not, any test that compares it against an object that isn't cached is moot (as both object may not be cached).

Does caching really improve things?

I don't think that "reconsider using X" is a very useful answer to "how do I test X happened".

For the consumer of a business object API, caching is just another way of implementing the same API. However, if your caching is implemented as a service to be used by many objects, then those who use it will see the caching behavior itself as the boundary of the black box. They may not care whether you use arrays or hashes to implement it, but they will care that the promised algorithm does what you claim it does. For example, if you have an algorithm that says that cache members should be polled for last use timestamp each time an object is fetched, then you can test for compliance with that specific contract.

I've read the above paragraph a couple of times. I still have no clue whether you're trying to make a point, or whether you're just stringing words together.

Comment on Re^2: Testing objects that cache

Replies are listed 'Best First'.
Re^3: Testing objects that cache by ELISHEVA (Prior) on May 15, 2009 at 15:46 UTC
No, not really. Once the OP is that far, his problem is solved. The claim that two objects behave the same (performance asside) is a testable claim. Therefore it should be tested. I don't think that "reconsider using X" is a very useful answer to "how do I test X happened". Presumably if we are implementing caching then caching does make a difference. Therefore a non-cached and cached implementation of the same object ought to have a statistically different performance profile. That is an assertion that can be tested. If no performance difference is observable, then we might wonder if indeed caching is happening. I've read the above paragraph a couple of times. I still have no clue whether you're trying to make a point, or whether you're just stringing words together. Unfortunately such a sarcastic remark does little to help me be more clear. What didn't you understand? Perhaps a more concrete example would help? Suppose I have a Person List API. I am responsible for programming a customer analysis report using that API. As the report writer I don't really care whether caching does or does not take place. I only care that the API provides whatever information I need to select the top 10 customers of the month in what my client subjectively feels is a reasonable amount of time. Testing performance time is OK, but testing specifically for caching is not. It violates the black box principle. How I get the performance doesn't matter - neither to my customers nor to me. Now suppose my customers don't like the performance. After doing some profiling I decide that the slow performance is due to too much paging. I decide to reduce the memory consumption of the Person List by adding caching. Not wanting to reinvent the wheel I look at several different caching libraries - some optimize for retrieval time, some optimize for memory usage. If the API lets me set the memory consumption of the cache , I care very much that the cache API does not exceed that memory threshold. I would want that claim tested. Caching is still an implementation detail. However, testing the caching library API to make sure it conforms to its promises does not violate the black box principle, because the API being tested is the caching behavior itself. Once I am done installing the code that uses the caching library, I go back to testing the customer analysis report. Once again I don't care about caching. I only care that my changed implementation seems fast enough to my customers. It really doesn't matter why it is faster, only that it is faster. Best, beth	[reply]

Replies are listed 'Best First'.

Re^3: Testing objects that cache
by ELISHEVA (Prior) on May 15, 2009 at 15:46 UTC

No, not really. Once the OP is that far, his problem is solved.

The claim that two objects behave the same (performance asside) is a testable claim. Therefore it should be tested.

I don't think that "reconsider using X" is a very useful answer to "how do I test X happened".

Presumably if we are implementing caching then caching does make a difference. Therefore a non-cached and cached implementation of the same object ought to have a statistically different performance profile. That is an assertion that can be tested. If no performance difference is observable, then we might wonder if indeed caching is happening.

I've read the above paragraph a couple of times. I still have no clue whether you're trying to make a point, or whether you're just stringing words together.

Unfortunately such a sarcastic remark does little to help me be more clear. What didn't you understand?

Perhaps a more concrete example would help? Suppose I have a Person List API. I am responsible for programming a customer analysis report using that API. As the report writer I don't really care whether caching does or does not take place. I only care that the API provides whatever information I need to select the top 10 customers of the month in what my client subjectively feels is a reasonable amount of time. Testing performance time is OK, but testing specifically for caching is not. It violates the black box principle. How I get the performance doesn't matter - neither to my customers nor to me.

Now suppose my customers don't like the performance. After doing some profiling I decide that the slow performance is due to too much paging. I decide to reduce the memory consumption of the Person List by adding caching. Not wanting to reinvent the wheel I look at several different caching libraries - some optimize for retrieval time, some optimize for memory usage. If the API lets me set the memory consumption of the cache , I care very much that the cache API does not exceed that memory threshold. I would want that claim tested. Caching is still an implementation detail. However, testing the caching library API to make sure it conforms to its promises does not violate the black box principle, because the API being tested is the caching behavior itself.

Once I am done installing the code that uses the caching library, I go back to testing the customer analysis report. Once again I don't care about caching. I only care that my changed implementation seems fast enough to my customers. It really doesn't matter why it is faster, only that it is faster.

Best, beth

[reply]