gulden has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I need your help to implement something to monitor CPU/MEM usage during or at the end of the execution of a perl program.

Imagine that a program that runs every N minutes which is launched by another Perl program, the execution of the latter can vary from a few seconds to 1 minute. How can I get CPU/MEM usage that result from executing this program.

some of the requirements are:

I've thought in some approaches:

  1. Implement my own Devel::MyProfiler and run the program with -d "option". (ex: $ perl -dMyProfiler program.pl)
  2. Implement a wrapper that get stats every N seconds from /proc/pid or "$ ps aux", and gets the average stats at the end of the program
  3. Implement a parallel program that makes a "$ ps aux" every N seconds
  4. Using a process trace tool!!
  5. ???

Any tip will be helpfull!!!

Note: this question was also posted in http://stackoverflow.com/questions/6372093/my-perl-program-stas

Tks,

gulden

«A contentious debate is always associated with a lack of valid arguments.»

Replies are listed 'Best First'.
Re: My Program Stats
by davido (Cardinal) on Jun 16, 2011 at 15:20 UTC

    Please make sure when you cross-post that you identify such, so that people here (or there) don't end up working on a problem that is solved. Identifying cross-posting will promote collaborative progress too. Getting a bunch of people doing research for you in parallel without knowing about each others' progress defeats the collaboration aspect of public forums.

    http://stackoverflow.com/questions/6372093/my-perl-program-stas


    Dave

      sorry, and tks for the advice.
Re: My Program Stats
by BrowserUk (Patriarch) on Jun 16, 2011 at 16:25 UTC

    Can I ask what your purpose for this is?

    You mention "average stats", but if, for example, your purpose is capacity planning, then averages are a poor measure. If the average usage for a process is say 2GB, you might consider that provisioning servers or VMs with 4GB is sufficient, but that might mask the fact that for some brief period, it required 5GB with the inevitable consequences.

    For the Windows platform, there are a whole bunch of performance monitoring counters and 2 or 3 different ways of accessing them that can provide CPU and memory usage statistics on a per process basis. They can provide instantaneous, average and lifetime peak stats. Of course, that doesn't help you on other platforms, but this type of thing will always need platform specific solutions.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      From what you said i can do:

      1. Launch MyWrapper (my_wrapper.pl)
      2. In MyWrapper: launch MyProgram (my_program.pl) using for instance Proc::Background.
      3. In MyWrapper: collect stats from Win32PerfCounters during or at the end of the execution of MyProgram

      This seems to me a valid solution for Win32.

        You might find Corion's DBD::WMI a suitable way of accessing the required information.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: My Program Stats
by flexvault (Monsignor) on Jun 17, 2011 at 16:59 UTC

    I've used this script (refined many times) with success. I don't use it for time, but checking the man for 'ps' you can get that info also. I use it from the parent to check on possible runaway child processes. If the ( current size - original size ) > max size, then I kill the child and fork a new child. (Note: the child may actually be working correctly, so the 1st kill is to ask the child to take an early quit, and if that doesn't work, I do a system kill.)

    You need to test how the kill is handled on different operating systems, but in general it's fast and works. I've used this on AIX, Linux ( Debian, openSuse, Ubuntu, and probability others ). You don't need the $Debug test, but I left it in to give you some ideas on debugging help. The 'Die_Rtn' can be replaced with a 'die', but I do "die" differently depending on whether it's needs a terminal or web response. You could also put the open in an 'eval'.

    Consider it a start, since the parent needs to know what operating system to consider for the max size. There are large differences between the operating systems, and also 32bit and 64bit environments.

    # How it's called my $NAME = "myperlpgm.plx"; my ($pmem1,$pmem2) = Display_Mem_Usage($$,$NAME); sub Display_Mem_Usage { # VSZ is size in KBytes of the virtual memory ( VSZ * 1024 ) # RSS is size in pages of real memory ( 1024 * RSS ) my $cpid = shift; my $name = shift; my $var = ""; my $fh; if ( $Debug >= 3 ) { $0 = "$name: Checking. . . "; } my $arg = qq| -o "vsz rssize" -p $cpid|; open ( $fh, "-|", "$PS $arg" ) or Die_RTN("Prefork: Not open \'$PS +\': $!"); while (<$fh>) { $var .= $_; } close $fh; my $rno = my @ref = split(/\n/,$var); if ( $rno < 2 ) { return ( -1, -1 ); } my $info = join(" ", split " ", $ref[1]); my ($vmem,$rmem) = ( split(/\ /,$info) ); ## { syslog('info',"Display_Mem_Usage: $PS $arg|$vmem|$rmem|$va +r"); } if ( $Debug >= 3 ) { syslog('info',"Mem_Usage: |$vmem"."K,$rmem"."K|$var"); } return ( $vmem * 1024, $rmem * 1024 ); }

    Hope it helps!

    "Well done is better than well said." - Benjamin Franklin

Re: My Program Stats
by locked_user sundialsvc4 (Abbot) on Jun 16, 2011 at 12:55 UTC

    What you are looking for (in general, not just with regards to Perl) is called a “profiler.”

    Profilers employ operating-system instrumentation hooks that are designed for this purpose.   And, they try to do this while affecting the behavior of the program as little as (or, as predictably as) possible, so that the results obtained under profiling will be comparable to those without.

Re: My Program Stats
by locked_user sundialsvc4 (Abbot) on Jun 16, 2011 at 21:07 UTC

    A much better strategy, overall, is to establish some kind of benchmark goal for your program, and then measure how often the program (does | does not) meet that goal.

    For example:   “All requests sent to this program will be replied-to within 0.1 seconds.”

    The advantage of a metric like this is that it is (a) practical, and (b) taken from the client’s point of view.   A given client does not care how many other clients you have.   But he or she will change their bookmarks forever (heh...) if they receive less-than-stellar service at any time.   (“Bad news has many brothers; Good news stands alone.”)   The data points that you will collect are purely, “pass or fail.” And that is generally good, because if your car missed the turn and smashed into the tree, it really does not matter if you missed the curve by five degrees vs. ten.   You (or, I should say, your next-of-kin) also have zero interest in what was the engine-temperature at the precise moment of (ugh...) impact.

    Another big advantage of such a strategy is that you can accurately determine the answer from log files ... i.e. “ex post facto.”   And it can be subjected to all sorts of statistical analyses, again, at your leisure ... after the fact.

      I understand your point of view and even agree, but I want to know what impact my programs are having in terms of CPU/MEM in the remote servers.