At my job, we are very often extracting data from seven large databases, each database having eight sub-databases. This is not like just table dumping, there is a lot of business logic in this extraction process, so that the process is heavily IO-bound, but also in part CPU-bound. These extraction processes are very long: from 4 to 8 hours for most, up to 3 or 4 days for a couple of them for the full data extraction to complete (and yes, we are extracting a very large volume of data).
What we do is to launch 7 * 8 = 56 processes through a queuing system and maintain the maximum number of active processes at a certain level, the other processes are just pending doing nothing until one slot becomes free for one of them. We have 4 CPU on our server. We found that, usually, the optimal number of processes running concurrently is somewhere between 8 and 12. (We have about 50 different data extraction applications, some are more heavily IO-bound than others, so that the optimal number of processes will vary to a certain extent with the nature of the extraction being run.)
Less than 8 processes in parallel, and the server appears to be underutilized (although we are doing a few other things on this server, it is really essentially dedicated to these heavy extractions tasks). More than 12 processes, and it appears that the overhead of context-switches starts to slow down the overall execution performance (the processes in themselves are not very memory-intensive, but there could be some underlying data caching, buffering and pre-fetching leading to a real memory consumption higher than what we think).
Anyway, in view of that, we usually set the queue to a maximum of about 10 processes running in parallel for our 4-CPU server.
In reply to Re: Useful number of childs revisited [SOLVED]
by Laurent_R
in thread Useful number of childs revisited [SOLVED]
by karlgoethebier
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |