http://qs1969.pair.com?node_id=1147873


in reply to Re: How can I limit the number of Cores in multi-process programming
in thread How can I limit the number of Cores in multi-process programming

Hi,

Thanks a lot for your reply. In my environment, many users are sharing hosts, and each job has CPU quota. If you request more quota like occuping the host exclusively, pending time would be longer, so I prefer to limit my CPU quotas to a optimal size.

  • Comment on Re^2: How can I limit the number of Cores in multi-process programming

Replies are listed 'Best First'.
Re^3: How can I limit the number of Cores in multi-process programming
by BrowserUk (Patriarch) on Nov 17, 2015 at 06:06 UTC
    In my environment, many users are sharing hosts, and each job has CPU quota.
    1. Is it a virtualised environment?

      Ie. Is your process running in a VM or container?

    2. What units are your CPU quotas specified in?
    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

      It's load balanced farm. User requests hosts with max_cpu specification, if your jobs use more cpu cores, would get warnings and the job would be killed.

      Thanks

[reply]

        Have you had a look at Parallel::ForkManager? It can be easily used to move from unbounded fork() to running with a limited number of children.

[reply]
        It's load balanced farm.

        Then you are going to have to either: a)limit the number of processes you run to match the number of cpu's you are allowed; or b) ask the operations/support people for that system if there is an API available for your purpose.

        In a single cpu, multicore system; any processes can and will be scheduled to run on the next available processor and it may run on a different processor for each timeslice. This is entirely transparent to the user-level code and it makes it pretty much impossible for application code to measure which and how many cores it is using at any given time. The solution to the problem on these types of system is to use an OS api that sets a mask for those cpus this process (or thread) is allowed to be scheduled to. On windows this is SetProcessAffinityMask(); there is probably a similar api available on modern versions of *nix, though I don't think it is a part of the POSIX spec.

        But, on a processor farm, the OS itself doesn't manage the allocation of processes to cpus, and each of those CPUs probably have multiple cores; so you'll need an API provided by the load-balancer to provide you with the control you need.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re^3: How can I limit the number of Cores in multi-process programming
by SuicideJunkie (Vicar) on Nov 17, 2015 at 14:09 UTC

    It sounds strange that you want to multithread for more speed, but then turn around and want to use less CPU % (so it takes longer again?).

    Do you have busy loops wasting CPU cycles or something? If so, use a sleep or a select(undef,undef,undef,0.01 to reduce the polling rate and drop the CPU usage to <1%.

    You can try a nap of a few milliseconds per iteration on your big long running loops as well if they're hogging resources, but if the loop is doing productive things, letting it finish is usually a good idea ;).

[reply]
[d/l]

In Section Seekers of Perl Wisdom