Are you saying that each process is launching several threads? You would have to explain that, but that sounds like defeating the purpose of Parallel::ForkManager. There are a number of technical differences between processes and threads, but they are essentially the same types of animal: my choice of words might be poor, but both are a stream of code execution. If you have a 8 processes running on 4 CPUs (or a CPU with four cores), you will have context switches, i.e. your program will switch from one process to another many times per second. Context switch has a cost (CPU registers, caches and execution stacks have to be saved and loaded) again, but that's often OK because if a process is waiting, say, for data from the disk, another process might do many things meanwhile before the first one is ready to continue. A thread is essentially a lightweight process, which means that context switch is likely to be less costly (depending on the operating system, it might actually be almost as expensive). Anyway, the basic idea of something like Parallel::ForkManager is to limit the number of processes in order to avoid too many context switches. But if each process can run many threads, then the point of the operation is essentially lost.
In my experience (which has nothing to do with Windows), it can be useful to run more processes than the number of CPUs. With our relatively IO intensive processes, the best performance results are obtained when you have about twice more processes than the number of CPUs. So, with our 4-CPU server, we usually run 8 to 10 processes in parallel, sometimes a bit more because some specific processes are just waiting for the other to complete and are sleeping most of the time. I can't extrapolate anything to the Windows OS, but I would assume that it is reasonable to accept a few more processes than the number of CPUs or CPU cores. Say anywhere between the number of cores and twice the number of cores. Only testing will tell you, because there are so many things out of your control (specific characteristics of your program, hardware and OS caching strategies, etc.) that it is almost impossible to predict what is going to happen unless perhaps you have a really deep thorough knowledge on your hardware and OS.
In brief, using multiple processes and multi-threaded processes sounds to me like a bad idea in general, but, to tell the truth, if you can guarantee that each process is generating, say, only two threads, you might end up closer to the optimum. But if each process is generating dozens of threads, then you are much more likely to be very far from the optimum.
In reply to Re: Parallel::ForkManager and CPU usage?
by Laurent_R
in thread Parallel::ForkManager and CPU usage?
by janmartin
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |