I don't see anything obviously wrong with your code. A couple of suggestions for tracking it down:

• Add some debug statements to let you know what the children are doing -- When it becomes slow, are all of them still active?
• Are you running out of some resource, like memory?
• Using something like Benchmark to find where the slowness is being introduced.

This looks to me like it needs some straightforward debugging to figure out where your bottleneck is.