I really don't think the answers to these questions will make it any easier to answer my original question. Nevertheless, I'll do what I can...

I'm testing a server that translates documents from one human language into another (e.g. Arabic to English). The server communicates via SOAP so my test script starts up the server and then acts as a SOAP client submitting translation 'jobs'. The server accepts each job and when it is complete the client retrieves performance data, including the speed of translation in words per minute. We (meaning someone in the company who is not me) have determined that these results are inaccurate because in a single test run (which consists of approximately 7 jobs with slightly varying settings) memory and/or disk caching is making the later runs appear to be too fast. I am already stopping and restarting the server between each job, therefore this discrepancy is not due to any explicit caching that is done by the application itself.

Unfortunately, I am not able to post any code for this; I hope my explanation of the behavior of the script above will be sufficiently enlightening.