Very well... then I shall step in and try to earn some XP’s that go in the other direction ...

The most-successful analogy I’ve found for a good multi-threaded process is that of a fast-food restaurant.   First of all, everybody makes it out of the store alive at the end of the shift.   At times they have been busy-as-crazy and at times they have been staring at the label on a french-fry machine.   You can see that each worker’s workflows are based on queues... in and out.   Also, each one contributes to the order in some way, but the guy who’s cooking those fries isn’t the one putting them onto the plate.   Finally, each worker’s task can be isolated from any of the others, say, for training (debugging) purposes.

In such a system, you can never say precisely what the system will be doing at any particular moment, but you can say that the system is adaptive to whatever happens, that it is tunable in real-time by the manager on duty, and that its behavior can be predicted through stochastic modeling.

There are more-elaborate variations on this, including the “jack of all trades model,” in which any worker can do any task (can play any role ...) and they switch hats according to which unit-of-work they select from which queue.   In this model, the number of workers is strictly an expression of the “maximum multiprogramming level” of the system; of the maximum amount of simultaneous activity that the system will currently attempt to perform under worst-case conditions.   The workflow system is literally a system of queues and the workers are merely the ones who are buzzing around it, making honey.

In every case:   each worker sleeps (consuming no CPU resources) until it has something to do.   While working on its appointed or selected unit of work, it doesn’t have any dependencies upon any other worker.   (If the work shows up in its queue now, that unit of work can be completed now.   It won’t have any lock-contention except between it and another worker who is now performing the same role, which won’t happen.)

For debugging purposes, as I said, it really comes down to design.   If the worker roles are laid out in this way, every worker can be pulled into a Test:: jig and it can be debugged in isolation.   The “plumbing” consists of stock CPAN parts, already debugged.   Workflow managers can be found there, too.