Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: How can I limit the number of Cores in multi-process programming

by BrowserUk (Patriarch)
on Nov 17, 2015 at 02:50 UTC ( #1147871=note: print w/replies, xml ) Need Help??


in reply to How can I limit the number of Cores in multi-process programming

I hope my perl script can run faster, however it should not eat up all the CPU resouce.

Why shouldn't it use all the CPU resource? What are you saving it for?

Essentially what I'm saying here is that cpu resource is time; and time inexorably moves forward; you cannot save it for later; so unused CPU is essentially wasted cpu that you can never recover. Your processor should either be running as close to 100% as your workload permits; or it should be idle because you have nothing to do. Trying to restrain it to say 75% or whatever; doesn't achieve anything not even energy savings.

Perhaps what you are concerned with is that you want to ensure that some other, sporadic processes will have an opportunity to run concurrently with this workload when they need to?

If so, the proper way to deal with that is not to try an 'reserve' some cpu; but rather to run this workload at a lower priority than the sporadic workload(s) so that you make full use of the cpu for this workload when can; whilst ensuring that those sporadic workloads will get a look in when they need it.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re: How can I limit the number of Cores in multi-process programming

Replies are listed 'Best First'.
Re^2: How can I limit the number of Cores in multi-process programming
by milesjin (Initiate) on Nov 17, 2015 at 03:50 UTC

    Hi,

    Thanks a lot for your reply. In my environment, many users are sharing hosts, and each job has CPU quota. If you request more quota like occuping the host exclusively, pending time would be longer, so I prefer to limit my CPU quotas to a optimal size.

      In my environment, many users are sharing hosts, and each job has CPU quota.
      1. Is it a virtualised environment?

        Ie. Is your process running in a VM or container?

      2. What units are your CPU quotas specified in?

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.

        It's load balanced farm. User requests hosts with max_cpu specification, if your jobs use more cpu cores, would get warnings and the job would be killed.

        Thanks

      It sounds strange that you want to multithread for more speed, but then turn around and want to use less CPU % (so it takes longer again?).

      Do you have busy loops wasting CPU cycles or something? If so, use a sleep or a select(undef,undef,undef,0.01 to reduce the polling rate and drop the CPU usage to <1%.

      You can try a nap of a few milliseconds per iteration on your big long running loops as well if they're hogging resources, but if the loop is doing productive things, letting it finish is usually a good idea ;).

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1147871]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (4)
As of 2022-12-08 22:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?