Re: Testing methodology

If you want to test queue-size limits under plausible conditions set up more-than-two consumers and more-than-two producers with random rates of speed from cycle to cycle but some known to be faster/slower than the others. Provide some way for the process to know that it was or was not blocked during its last request (and test that code too). Let one thread launch this test case, wait for them to finish and then judge the scores (how much sent/received, how many times blocked due to empty or full) by some heuristic that is not absolute. Fast producers and fast consumers should have run against limits; slow consumers in the presence of (only) fast producers probably not. No outcome absolutely deterministic, but you can deduce whether the outcomes are plausible or not. Vary the above description according to exactly what your package is or is not designed to be capable of.

Comment on Re: Testing methodology

Replies are listed 'Best First'.
Re^2: Testing methodology by BrowserUk (Patriarch) on Mar 06, 2012 at 04:06 UTC
... set up more-than-two consumers and more-than-two producers with random rates of speed from cycle to cycle but some known to be faster/slower than the others. Thanks for the response anonymonk. With regard to trying to orchestrate indeterminacy. I've tried in the past and it is a sucker's game. The one sure-fire thing you learn about concurrency when you've done enough of it, is that you do not have to orchestrate deadlocks, live-locks, race conditions, or any of the other nasties. Run a bad system for a while and make sure plenty of other different things are happening in the same system, and the nasties will make themselves known. Hence, my surety against these anomalies is to run my test suite (posted elsewhere) with big numbers and then play music, watch the Iplayer, and defrag my hard disks concurrently. It is a fair bet that a more diverse set of inter-thread timings occurred during that than I could ever hope to orchestrate deliberately. If the test suite completes correctly with all that going on, it is probably bomb proof. A typical test run consists of this: `C:\test>perl async\Q.pm -N=1e6 -T=400 -SIZE=400 1e6 items by 400 threads via three Qs size 400 in 811.944000 seconds` [download] That's 1 million items fed from 1 thread via a queue to a pool of 200 threads; those threads feed it via a second queue to another pool of 200 threads; which in turn feed it via third queue back to the main thread. At the same time I'm listening to Division Bell, whilst "Racing for Time" (movie) plays away to itself (with the volume off) in a tab in my browser. All of which simply means that my 4-cores are averaging over 90% usage each and I don't need any heat in the room despite being close to zero outside because the cpu fan is running close to flat out. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l]

Replies are listed 'Best First'.

Re^2: Testing methodology
by BrowserUk (Patriarch) on Mar 06, 2012 at 04:06 UTC

... set up more-than-two consumers and more-than-two producers with random rates of speed from cycle to cycle but some known to be faster/slower than the others.

Thanks for the response anonymonk.

With regard to trying to orchestrate indeterminacy. I've tried in the past and it is a sucker's game.

The one sure-fire thing you learn about concurrency when you've done enough of it, is that you do not have to orchestrate deadlocks, live-locks, race conditions, or any of the other nasties. Run a bad system for a while and make sure plenty of other different things are happening in the same system, and the nasties will make themselves known.

Hence, my surety against these anomalies is to run my test suite (posted elsewhere) with big numbers and then play music, watch the Iplayer, and defrag my hard disks concurrently. It is a fair bet that a more diverse set of inter-thread timings occurred during that than I could ever hope to orchestrate deliberately. If the test suite completes correctly with all that going on, it is probably bomb proof.

A typical test run consists of this:

C:\test>perl async\Q.pm -N=1e6 -T=400 -SIZE=400
1e6 items by 400 threads via three Qs size 400 in 811.944000 seconds
[download]

That's 1 million items fed from 1 thread via a queue to a pool of 200 threads; those threads feed it via a second queue to another pool of 200 threads; which in turn feed it via third queue back to the main thread. At the same time I'm listening to Division Bell, whilst "Racing for Time" (movie) plays away to itself (with the volume off) in a tab in my browser. All of which simply means that my 4-cores are averaging over 90% usage each and I don't need any heat in the room despite being close to zero outside because the cpu fan is running close to flat out.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

[reply]
[d/l]