I will talk of OS heavyweight shell processes, not threads, and certainly not Parallel::ForkManager. It even does not have much to do with Perl (although Perl is used in our programs). But I still believe that this is not off-topic and actually quite relevant.

At my job, we are very often extracting data from seven large databases, each database having eight sub-databases. This is not like just table dumping, there is a lot of business logic in this extraction process, so that the process is heavily IO-bound, but also in part CPU-bound. These extraction processes are very long: from 4 to 8 hours for most, up to 3 or 4 days for a couple of them for the full data extraction to complete (and yes, we are extracting a very large volume of data).

What we do is to launch 7 * 8 = 56 processes through a queuing system and maintain the maximum number of active processes at a certain level, the other processes are just pending doing nothing until one slot becomes free for one of them. We have 4 CPU on our server. We found that, usually, the optimal number of processes running concurrently is somewhere between 8 and 12. (We have about 50 different data extraction applications, some are more heavily IO-bound than others, so that the optimal number of processes will vary to a certain extent with the nature of the extraction being run.)

Less than 8 processes in parallel, and the server appears to be underutilized (although we are doing a few other things on this server, it is really essentially dedicated to these heavy extractions tasks). More than 12 processes, and it appears that the overhead of context-switches starts to slow down the overall execution performance (the processes in themselves are not very memory-intensive, but there could be some underlying data caching, buffering and pre-fetching leading to a real memory consumption higher than what we think).

Anyway, in view of that, we usually set the queue to a maximum of about 10 processes running in parallel for our 4-CPU server.

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.