Some examples.
1. I have one function that always caches its results. It does a very expensive approximate pattern match on a DB and saves the result. How much memory it uses for this is documented (and directly related to DB size) and user can't turn it off. In a typical app, performance difference is like 5-10,000x between first and second call. Testing is easy as I just have to verify that consistent results are coming back and the performance difference is so huge that it is easy to see. The database is read only for each run, so this is easy to implement (no need to fiddle with "dirty pages", etc.

2. I have another function that is not strictly "caching" as it precomputes a whole bunch of things that I think that you are going to ask, before you ask them (actually makes it more efficient to answer a whole bunch of questions at the expense of a lot of horsepower at the beginning). This is an example of a "cooperative" thing. You have to tell me that this dataset is special via a "study" call. If you do that intelligently then overall performance speeds up dramatically. The normal api calls aren't changed. So testing is working with and without "studying" this dataset first.

3. There are lots of examples where caching although transparent to the user API, the user nevertheless knows that caching is going on. One example would be file I/O buffering, say to stdout. $|=1 turns this off.

4. If you are caching results for something that is dynamic (data changes during execution of your program), that is usually something more appropriate for the module to mess with, not "something between you and it".

5. Almost always it is not wise to cache something that is already being cached by the O/S. An example of this would be file I/O. I have one app with a large number of files >500 and after they've all been looked at once, they become memory resident due to O/S caching. I don't have to do anything. If I tried to cache these files, performance could go down as this could reduce amount of space that the O/S has to do this function more efficiently than I can.

6. I personally have no problem adding api calls that would tell the user some stats on how well the caching is working. These would be extra optional things. For example the Perl hash allows you to query numBuckets and Buckets used. So what? You don't have to know this to use it, but if you are curious, you can find this out.

7. I also have no problem with "private" calls that look at the "guts" and report how things are going. I don't export these methods or functions. Of course any user can call these with fully qualified name, at their own risk.

8. I would have some sort of test() method that is not exported to be used for module verification.

So, I see as separate things:
A) How to test that this module is actually working? This is for developer and build process.
B) How well is this caching gizmo working for me? This is mainly based upon: consistent results and measured performance differences.

sorry that I was so long winded....

The first thing you probably want to test is that the cached object and the non-cached object in fact do have the same "black box" behavior.

His problem is "how do I test whether caching was used"? If you don't know an object you have is a cached object or not, any test that compares it against an object that isn't cached is moot (as both object may not be cached).

Does caching really improve things?

For the consumer of a business object API, caching is just another way of implementing the same API. However, if your caching is implemented as a service to be used by many objects, then those who use it will see the caching behavior itself as the boundary of the black box. They may not care whether you use arrays or hashes to implement it, but they will care that the promised algorithm does what you claim it does. For example, if you have an algorithm that says that cache members should be polled for last use timestamp each time an object is fetched, then you can test for compliance with that specific contract.

No, not really. Once the OP is that far, his problem is solved.

The claim that two objects behave the same (performance asside) is a testable claim. Therefore it should be tested.

I don't think that "reconsider using X" is a very useful answer to "how do I test X happened".

Presumably if we are implementing caching then caching does make a difference. Therefore a non-cached and cached implementation of the same object ought to have a statistically different performance profile. That is an assertion that can be tested. If no performance difference is observable, then we might wonder if indeed caching is happening.

I've read the above paragraph a couple of times. I still have no clue whether you're trying to make a point, or whether you're just stringing words together.

Unfortunately such a sarcastic remark does little to help me be more clear. What didn't you understand?

Perhaps a more concrete example would help? Suppose I have a Person List API. I am responsible for programming a customer analysis report using that API. As the report writer I don't really care whether caching does or does not take place. I only care that the API provides whatever information I need to select the top 10 customers of the month in what my client subjectively feels is a reasonable amount of time. Testing performance time is OK, but testing specifically for caching is not. It violates the black box principle. How I get the performance doesn't matter - neither to my customers nor to me.

Now suppose my customers don't like the performance. After doing some profiling I decide that the slow performance is due to too much paging. I decide to reduce the memory consumption of the Person List by adding caching. Not wanting to reinvent the wheel I look at several different caching libraries - some optimize for retrieval time, some optimize for memory usage. If the API lets me set the memory consumption of the cache , I care very much that the cache API does not exceed that memory threshold. I would want that claim tested. Caching is still an implementation detail. However, testing the caching library API to make sure it conforms to its promises does not violate the black box principle, because the API being tested is the caching behavior itself.

Once I am done installing the code that uses the caching library, I go back to testing the customer analysis report. Once again I don't care about caching. I only care that my changed implementation seems fast enough to my customers. It really doesn't matter why it is faster, only that it is faster.

Best, beth