in reply to To Fork or Not to Fork. Tis a question for PerlMonks

pimperator:

Programs not waiting for user input are dominated by CPU or I/O. If your processing is CPU bound, then splitting up the among the available processors can help. Similarly, if your processing is I/O bound, splitting your work over multiple I/O devices can speed things up significantly. However if your task is I/O bound and you can't split the work over multiple I/O devices, then splitting the work among several processes can make things go even slower.

Keep in mind that I/O devices include the network as well as disk drives. I've had a couple jobs where file processing was limited by network bandwidth (the disk drives were in a SAN), so we overcame the bottleneck by splitting the job over several computers.

I suggest you first find out what your bottleneck is, and then think about an appropriate strategy to split the work. In the (very rare) case that the non-dominated resource is still heavily used (e.g. you're using 100% of the I/O and 90% of the CPU), then splitting the work up may not gain you much, as you'll almost immediately hit the next bottleneck. Again, this isn't a common case.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

  • Comment on Re: To Fork or Not to Fork. Tis a question for PerlMonks

Replies are listed 'Best First'.
Re^2: To Fork or Not to Fork. Tis a question for PerlMonks
by pimperator (Acolyte) on Jul 01, 2014 at 16:52 UTC

    thank you for the reply. I imagine the computer's bottleneck is the I/O so I'll only do parallel computing for the CPU processes