contact_chatan has asked for the wisdom of the Perl Monks concerning the following question:

Issue: our parser is using 100% CPU once run on a machine, till it finishes its processing. Now I have to increase the efficiency and I have came up with two solutions:

Instead of reading one line at a time from the log file, I want to read a block and keep it in memory (lets say 10 kb) , However we are facing some problem then how our parser will react as it read file line by line instead of chunk of file and in this case how will I parse our log file.

Perl has a Win32 package. How can I use this package to reduce the CPU usage as I got the following line of code but I am unable to use it.

As it gives cpu time always 0. I am working on windows platform: (I think we have the problem in red as it may be for linux then how can I use this code for windows.)

Link of code is: resource control: CPU 

CPU_start(); CPU_max(30) while 1;
 

 

use Time::HiRes qw(time);
 
sub CPU_used {
    (map {$_->[13]+$_->[14]} 
        [split " ", Cat("/proc/self/stat")])[0]
}
 
{ my %start = (tm => 0, cpu => 0);
sub CPU_start { @start{"tm","cpu"} = (time(),CPU_used()) }
sub CPU_max {
    my ($max, $real, $cpu) = ($_[0], time()-$start{tm},
        CPU_used()-$start{cpu});
    return unless defined($max) and $max > 0;
    &sleep( $cpu/$max-$real );
}}
 
#
# macro used from CPU_used() and CPU_max()
#
 
sub sleep { select undef,undef,undef,$_[0] }
sub Cat {
    local *F;
    open F, "< ".$_[0] or return;
    local $/ unless wantarray;
    return <F>;
}

[download]
Seeking a great help from your side. Please do it as soon as because it is very urgent for me.

Thanks

Chatanya Agrawal

20050224 Edit by ysth: code and p tags; id://-ify link

Replies are listed 'Best First'.
Re: CPU utilization
by ysth (Canon) on Feb 24, 2005 at 11:38 UTC
    Presumably you have a loop reading the log file like: 
    while (my $logline = <LOG>) {
    # parse it
}

    [download]
    Making a process max out at a certain percentage of CPU is not easy to do (because it's generally not a useful thing to do; if the CPU isn't otherwise used, why not use 100%? and if there are other things running, setting different priorities works out better) especially I think on Windows. It may be sufficient for your purposes just to put in a line like sleep 1 unless $. % 100 which makes perl stop for at least a second every 100 lines.

    However, you may be able to get the linux code you posted to work by getting rid of sub Cat entirely and making CPU_used like: 

    sub CPU_used {
   (map $_->[0]+$_->[1], [times])[0]
}

    [download]
    But unless I'm totally misunderstanding it, you should be using CPU_max(.2) to get a max of 20% CPU use.

    Update: note that calling CPU_max every iteration is counterproductive; I'd figure out how many lines you process per second and make your code: 

    CPU_start();
while (my $logline = <LOG>) {
    # parse it
    CPU_max(.2) unless $. % $lines_per_second;
}

    [download]
    (using your lines per second figure, and adjusting if needed for finer granularity.)
[reply]
[d/l]
[select]
      Hi, Its not working as it is taking 100% cpu utilization Thanks , Chatanya Agrawal
[reply]
        Is this ActiveState's perl? Which windows operating system are you using? Just before the sleep call in CPU_max, try putting in a 
        print "CPU_max: cpu: $cpu  real: $real\n";

        [download]
        and run it for a little while to see how it goes.

        You might also try getting rid of the sleep function and instead putting use Time::HiRes "sleep"; at the top of your code.

[reply]
[d/l]
[select]
Re: CPU utilization
by holli (Abbot) on Feb 24, 2005 at 11:20 UTC
    How can I use this package to reduce the CPU usage as I got the following line of code but I am unable to use it.
    You canīt. Win32 is not magic wand you wave to reduce your CPU usage. You can, however, increase the efficiency of your code.
    Please describe what you want to do with your logfile, and please provide some sample data.

    And, please put <code></code>-tags around your code, to increase the chance that one will read it.

    Update: code-tags have been added while writing this post.
    Update: for linux/unix there is utility named "nice" that you can use to reduce cpu-load of a program. (with the price it will take longer to run).


    holli, /regexed monk/
[reply]
Re: CPU utilization
by Anonymous Monk on Feb 24, 2005 at 11:36 UTC
    Issue: our parser is using 100% CPU once run on a machine, till it finishes its processing.
    Good for you! I wish my programs could use 100% CPU time until they are finished. Unfortunally, they usually have to compete against other programs, and don't get enough 100% of the CPU time. 100% CPU time is (usually) a good thing.
    Instead of reading one line at a time from the log file, I want to read a block and keep it in memory (lets say 10 kb) , However we are facing some problem then how our parser will react as it read file line by line instead of chunk of file and in this case how will I parse our log file.
    Perhaps this will make your program more efficient. Perhaps not. It all depends on what your program is doing. If I had to bet though, I'd bet that it wouldn't make your program more efficient. If you're using 100% CPU time, it seems your program is CPU bound, and fiddling with size of the chunks you read from disk matters mostly from programs that have an I/O bottleneck.
    Perl has a Win32 package. How can I use this package to reduce the CPU usage as I got the following line of code but I am unable to use it.
    What line of code is that, and why are you unable to use it?
    Seeking a great help from your side. Please do it as soon as because it is very urgent for me.
    You really think that going to a forum with volunteers of various quality is a smart thing to do if you have an urgent problem?
[reply]
Re: CPU utilization
by jbware (Chaplain) on Feb 24, 2005 at 12:20 UTC
    Obviously efficency is important, but if a concern is about your script hogging the CPU when its running, have you considered a non-perl solution? Namely, lower the priority of the process running. It won't run faster, but at least other processes won't be impacted as much by your script. Go to the command line and type: start /? for info on how to start w/ a lower priority. Note, most things start with NORMAL, so using /LOW or /BELOWNORMAL should give your other processes some breathing room.

    -jbWare
[reply]
Re: CPU utilization
by inman (Curate) on Feb 24, 2005 at 13:49 UTC
    When you read a file 'line by line' in your Perl script, your computer is using an IO buffer to read a suitable chunk of file into memory in the background. The mechanics of the system vary between OS's but the idea is to optimise the IO read operations. Reading a chunk of a file into a memory (using a binary read operation) and then parsing it line-by-line introduces an extra step and therefore decreases the speed of the application.
[reply]
      Hi inman,
      I am agree with you. So what is the solution of this problem?
      Can it be solved or not? Or its h/w dependent
      Chatanya
[reply]

Back to Seekers of Perl Wisdom