mattr has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I'm designing a cross-platform application that will be able to fork processes but I don't want it to be a big CPU eater. Anybody have a good way to do so reliably?

CPAN says Sys::Load fails on Windows quite a bit, though there might be something accessible by Win32::API. Now I'm thinking of checking how long it takes to execute a command as processes are added but I'm not sure this will reflect actual niceness wrt other apps. Any experience here?

One option I suppose would be to give up and just specify a maximum number of processes, like apache does in its config file. Or some kind of concurrent scheduler for which you can dial the relative niceness. But I'd rather have as little configuration and as much intelligence as possible. Pure perl would also be a plus.

The end result would automatically keep it down to a small number of processes on a slow machine (or maybe at a time when cpu load is high for some reason) but would not blink at launching many processes if resources are available. Thanks for your help.

Matt

Replies are listed 'Best First'.
Re: Checking system cpu load when forking?
by EvanCarroll (Chaplain) on Nov 09, 2005 at 02:47 UTC
    Some things that don't compute:

    • "cross-platform" AND Win32::API in the same paragraph
    • but I don't want it to be a big CPU eater. Anybody have a good way to do so reliably?

    I'm not even sure as to what your trying to do, Dynamic forking maybe? If so check out Parallel::ForkManager as a starting point, and fork, perlipc, perlfork. Did you even bother looking at Sys::Load's docs. It apears as if it only gets information, which is system specific not task specific. Could you describe how you would like that to help? Or maybe more of what your goal is? Essentially, if a task takes 500,000wallclocks to complete, it doesn't matter how you cut it, it is going to take at least 500_000wallclocks.


    Evan Carroll
    www.EvanCarroll.com
[reply]
      Hi, and thanks for the reply! To clarify..

      • The objective is that I want my app to work quietly without slowing down the system even if it is being run on slow hardware or on hardware the resources of which are already in high demand i.e. by virus checking, printing, etc. The question is how to tell how busy the machine is, and to do so reliably with a minimum of interaction and a minimum of hassle whether the machine is a windows 98 pc, a linux laptop, a mac, or some windows or linux server.
      • Sys:Load works on linux so I mentioned Win32::API with the idea that I might have to think up some alternate strategy to be used in case we are running on a windows machine. Which I don't want to do.
      • Yes, I've read the Sys::Load docs (a related page also says it is limited to systems with getloadavg() apparently) and it works fine on linux anyway. Haven't tried it anywhere else. I was hoping Sys::Load would help since it returns an average of cpu load for the last minute. This knowledge would let the software take action when it load is above 0.8. It might tell a user that it is not a good idea to add more concurrent jobs, or it might choose to prioritize, delay or offload some things. Wallclock doesn't tell you how hard the machine is working. It is more about useability and figuring out whether more resources should be added.
      • I do have some studying to do, though I have of course read about Parallel::Forkmanager which I plan to use, and POE which I would also like to use in places. I haven't used these modules yet though, and I am not so familiar with perlfork. (The links you provided about perlipc and perlfork are quite nice, I'll go through them, thank you). I hope this is enough information. It would be nice if there was a PP heuristic to figure out how much you are stressing a machine and also to derive a baseline for how much power the machine has so you can plan resource loading in advance. Also I don't want to rewrite a module or require tweaking even when moving between machines. However if necessary I'm willing to consider a collection of two or more platform-specific solutions to figuring out the cpu load (and maybe available bandwidth and disk space too)
[reply]
Re: Checking system cpu load when forking?
by Anonymous Monk on Nov 09, 2005 at 09:32 UTC

    Checking CPU load for a process (and all other system counters) from Perl is done with Win32::Perflib . Some examples are available around the monastery (perflib is not particulary easy) .

    Forking should be done with Win32::Job . This even gives you a status() method that has access to the accumulated time of the process, in kernel and user mode, which you could use to assess your processus CPU consommation (like buy doing a delta between polling times).

[reply]
      Thank you very much. I'm not too familiar with the objects that come back from Win32::Perflib but it seems to be a solution for NT and later windows systems. Also Win32::Job looks quite useful in eliminating zombies from plugins.

      Of course the drawbacks are they only work on windows, and not win98 which might need it the most. And I'm not sure if wxperl running from a pp archive will be able to spawn using the perl command like the example given. I'm still looking for a cross-platform heuristic but these modules will be a very useful tool. By the way, can perflib return the overall cpu load of the whole system, or only the process you specify? And it mentions different "levels". Would that let me find overall impact of all subprocesses by checking the load caused by the parent process? Thanks a bunch.

      Just for the sake of argument, is there a problem with using Forkmanager and perlfork as discussed above on windows? Also, the Job module says it is good because it will kill grandchild zombies. Doesn't perl work that way on unix? (and is the failure of perl to do that on windows a windows-specific related to not being able to fork correctly? Thanks again.

[reply]

        Win32::Process::Info is mush easier to use than Win32::PerfLib and can return (almost?) all the same information in a form that is much better structured. However, it will only work on win95/98(SE) if they have had the WMI upgrade added.

        That said, I'd have to wonder about the efficacy of writing new code to support a platform that is 8 years old and for which the manufacturer dropped support over 2 years ago.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        Actually what you will get for calling Win32::Perflib is a massive structure made of hash of hash of hash ... up to 7 levels. This structure contains everything that the performance monitor service of Win32 is able to collect, and that includes global cpu load (per CPU on a SMP system, but there maybe also a grandtotal) . Your problem will be to find where it is !

        I found that the perflib struct was almost impossible to understand so i had to test thing many times in order to get the info i wanted from it . Dave Roth has some example scripts on his "roth consulting" site, he wrote the "Win32 perl programing" New Riders book (which i don't recommand by the way, it's close to useless )

        I don't think you can assess the cpu impact of a job (that is a pool of process) through perflib, and you will still have to add the load of the different forked process.

        About forking without Win32::Jobs, the only thing i can say is: even IPC::Open3 didn't work very well for me and that's standard in the active state distribution ! You can't even pass filehandles to the forked process. My feeling is that if you want to run under windows you had better use the windows way ...

[reply]

Back to Seekers of Perl Wisdom