prad_intel has asked for the wisdom of the Perl Monks concerning the following question:

Hi ,

I have a script which goes through around 80,000 to 1,00,000 files or even more sometimes, during this i see that the processor usage goes to 100% and System Cache Memory usage keeps shooting up for every cycle of operation it does.

I also see that when I open and close a window the priority is given to the explorer.exe and perl haults for a few milli seconds thereby giving a sudden dip in the processor usage.

I am also planning to forcibly make it to sleep for a few seconds after 50,000 files have gone through , will this add a tinge to make it a non crashable program.

Its been great surfing this site and am planning i would spend more time here than with my girl-friend.

Thanks and Regards

Pradeep.S

Replies are listed 'Best First'.
Re: Processor and Cache Memory usage
by sasikumar (Monk) on Jan 21, 2005 at 06:19 UTC
    Hi

    I am also planning to forcibly make it to sleep for a few seconds after 50,000 files have gone through , will this add a tinge to make it a non crashable program.



    This not the solution. This does nothing other than sleeping and again wasting your time instead of your CPU time.

    Probabaly what you would require, is to change your algorithm if possible. Else change the way you read your files. ie read it in blocks so u would have little memory usage.

    Show us your code that could help us in determining an optimal solution.

    Also try to visit this page http://www.ccl4.org/~nick/P/Fast_Enough/ which can help you. Also dont forget see this node 422918 This has some expert comments that might help you

    Thanks
    SasiKumar
Re: Processor and Cache Memory usage
by Anonymous Monk on Jan 21, 2005 at 14:06 UTC
    Uhm, I fail to see what the problem is here. There's nothing wrong with a 100% CPU usage. In fact, 100% CPU usage is what you want. Anything less, and it means you've paid for expensive pieces of metal you aren't using. There's nothing wrong with cache usage either. In a modern OS, every process uses the cache. What you want is a lot of cache hits, and not a lot of cache misses. Unfortunally, Perl isn't very good in this department (due to 1) high memory usage, and 2) lots and lots of pointers). But that's the price you pay for the flexibility you get.
Re: Processor and Cache Memory usage
by robot_tourist (Hermit) on Jan 21, 2005 at 08:47 UTC

    Threads may be your friend here. Also, I don't know your code (not that I'd be able to help if I did :) and I don't know enough about Perl's internals, but you must be certain there are no memory leaks. The behaviour you see when switching windows is to be expected on Windows.

    Other solutions: faster PC or rewrite in a faster language.

    How can you feel when you're made of steel? I am made of steel. I am the Robot Tourist.
    Robot Tourist, by Ten Benson

Re: Processor and Cache Memory usage
by holli (Abbot) on Jan 21, 2005 at 13:12 UTC
    You might find this interesting.

    holli, regexed monk