in reply to Re^2: Help with multiple forks
in thread Help with multiple forks
Yes, “learn about pipes” because this would make your job a helluva lot simpler. Start a pool of child-processes that read from a pipe and do the work that has been given to them by means of that pipe. (When the pipe is closed by the writer, the children’s read-requests fail and when this happens they terminate themselves.)
Likewise, instead of “starting” the second-stage processes when the first stage has finished, have the first-stage processes write messages to a second pipe that is listened-to by the second-stage processes which are built using the same design. After the first-stage processes consume their work and die-off, the second-stage processes in turn consume their work and die, and so on, until the parent finally realizes that all of its children have died (as expected) and it then terminates.
Now, all of the processes (regardless of their role) do their initialization and termination only once, and perform their jobs as quickly as they are able, and the pipes take up the slack. You tune the behavior of the system for maximum throughput by tweaking the number of processes that you create, and they perform work at that constant rate no matter how full or how empty the pipes may be.
Think: production line.
“Over-committing” a system produces performance degradation that becomes exponential after a period of time in which it is linear, a harrowing phenomenon called (politely) “hitting the wall.” The curve has an elbow-shaped bend which goes straight up (to hell). For instance, I once worked at a school which needed to run computationally-expensive engineering packages on a too-small machine. If one instance was running, it took about 2 minutes; with five, about 4. But with seven, each one took about 18 minutes and it went downhill from there ... fast. A little math will tell you that the right way to get seven jobs done in 6 minutes (on average) is to allow no more than five to run at one time. It worked, much to the disappointment of the IBM hardware salesman. The rest sit in a queue, costing nothing for the entire time they sit there not-yet-started. Likewise, a queue-based architecture will consistently deliver x results-per-minute at a sustained rate even if there are larger-y pieces of work to be performed. A thread (or process) is not a unit-of-work.
Replies are listed 'Best First'. | |
---|---|
Re^4: Help with multiple forks
by Anonymous Monk on May 31, 2012 at 15:23 UTC | |
Re^4: Help with multiple forks
by Anonymous Monk on May 31, 2012 at 18:13 UTC |