http://qs1969.pair.com?node_id=1147882


in reply to Re^2: How can I limit the number of Cores in multi-process programming
in thread How can I limit the number of Cores in multi-process programming

In my environment, many users are sharing hosts, and each job has CPU quota.
  1. Is it a virtualised environment?

    Ie. Is your process running in a VM or container?

  2. What units are your CPU quotas specified in?
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re^3: How can I limit the number of Cores in multi-process programming

Replies are listed 'Best First'.
Re^4: How can I limit the number of Cores in multi-process programming
by milesjin (Initiate) on Nov 17, 2015 at 08:31 UTC

    It's load balanced farm. User requests hosts with max_cpu specification, if your jobs use more cpu cores, would get warnings and the job would be killed.

    Thanks

[reply]

      Have you had a look at Parallel::ForkManager? It can be easily used to move from unbounded fork() to running with a limited number of children.

[reply]

        Hi Corion,

        Thanks. Yes, I have checked the Parallel::ForkManager. Parallel::ForkManager is good enough for my taks. It can limit the number of processes, not CPU cores.

[reply]
      It's load balanced farm.

      Then you are going to have to either: a)limit the number of processes you run to match the number of cpu's you are allowed; or b) ask the operations/support people for that system if there is an API available for your purpose.

      In a single cpu, multicore system; any processes can and will be scheduled to run on the next available processor and it may run on a different processor for each timeslice. This is entirely transparent to the user-level code and it makes it pretty much impossible for application code to measure which and how many cores it is using at any given time. The solution to the problem on these types of system is to use an OS api that sets a mask for those cpus this process (or thread) is allowed to be scheduled to. On windows this is SetProcessAffinityMask(); there is probably a similar api available on modern versions of *nix, though I don't think it is a part of the POSIX spec.

      But, on a processor farm, the OS itself doesn't manage the allocation of processes to cpus, and each of those CPUs probably have multiple cores; so you'll need an API provided by the load-balancer to provide you with the control you need.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        Thanks a lot for your time. It's helpful.

[reply]

In Section Seekers of Perl Wisdom