SuicideJunkie:

When you've got too many tasks that need to be moved forward asynchronously, you might want to use a state-machine architecture.

One example: have perhaps a couple of threads per CPU. Each of them has their own task list. Then the thread just loops over the tasks, moving them ahead as they can. Whenever you're adding a task to the mix, just add it to the shortest task list. You might also have an event handler that when it sees particular events occur, it updates the information for the appropriate task. Then the thread can do any processing required the next time it examines the task.

In the tunnel-digging analogy, you might have a dozen people digging a four-person wide hole. Each one has a shovel and a bucket. One task would be to dig out a scoop of dirt and toss it into the bucket. Another task might be to carry full buckets out to the dumping area. So you've got a million "dig a scoop" jobs.

The first task(person) scoops out some dirt and tosses it in the bucket. Then he looks at the next task on the list: carry a full bucket to the dumping area. His current bucket isn't full, so that task stays on the list and he goes to the next "dig a scoop" tasks. He digs another scoop of dirt and puts it into the bucket. Eventually, he'll get to a "carry the bucket" task when his bucket is full, so he'll pick up the bucket and carry it to the dump. In the meantime, since a slot has opened up, one of the other threads waiting patiently for a place at the front of the line walks up and starts processing.

Alternatively, you might be slicing your tasks up too much. You might need to make your tasks larger so the thread can do more work without worrying about context switching. Using the tunnel analogy again, let's say there's only one task: "Dig as much as you can, fill the bucket, and carry the dirt to the dump". In this case, the person isn't cycling through the tiny tasks. Rather he'd have hundreds of tasks like:

while (!bucket_is_full()) {
    my $dirt = dig_a_scoop();
    toss_in_bucket($dirt);
}
move_to($dump);
empty_bucket();
[download]

...roboticus