smellysocks has asked for the wisdom of the Perl Monks concerning the following question:

Oh great and wise perl monks, forgive my foolishness if I failed to seek the answer to this question hard enough...

Currently I have a program processing a large amount of data, and my system with 2gigs of memory (virtual and real combined) is running out of memory while processing it (firewall logs -> Thanks Dcom worms!).

I'm sick of my perl program crashing when it ran out of memory. What I would like to do is to exit gracefully when 80% of the memory on the host has been utilized.

Anyone know what prayer I can sing that will let me calculate these values?

Adding hardware/virtual memory is not an option, and is sloppy IMNSHO. I want this to be able to scale up without any issues in the future.

Replies are listed 'Best First'.
Re: Remaining Memory on a system
by esh (Pilgrim) on Aug 29, 2003 at 14:54 UTC

    Assuming you're on Linux, you might consider Linux::MemInfo 

    #!/usr/bin/perl -w
use strict;
use Linux::MemInfo;

my %meminfo = get_mem_info();

my $total   = $meminfo{'MemTotal'};
my $free    = $meminfo{'MemFree'};
my $buffers = $meminfo{'Buffers'};
my $cached  = $meminfo{'Cached'};

my $really_free  = $free + $buffers + $cached;
my $percent_free = int(100 * $really_free / $total);

print <<"EOM";
Total: $total
Free:  $really_free ($percent_free%)
EOM

    [download]

    If you're not on Linux, you might consider sharing what OS you are using.

    -- Eric Hammond

[reply]
[d/l]
      Sorry. I should have mentioned that... I'm running this app on a Solaris 2.8 box on an E250.
[reply]
Re: Remaining Memory on a system
by Fletch (Bishop) on Aug 29, 2003 at 14:41 UTC

    Check perldoc perlvar for $^M and ask your sysadmin to configure more swap, but the better course of action is to figure out what about your approach is causing you to chew up all of your RAM to begin with.

    • If you're slurping entire files into memory at once, don't.
    • If you've got to keep intermediary results, consider writing them out to a DBM file or to an RDBMS rather than keeping everything in RAM.
    • Break your input data set into smaller chunks if possible, process the smaller chunks, then use those results to produce a final agregate result.
[reply]
[d/l]
[select]
      Check perldoc perlvar for $^M

      Yes, and? $^M only has a use after running out of memory, a situation that the OP wants to avoid. Furthermore, use of $^M only means you will run out of memory quicker, and Perl will just allocate a chunk of memory when $^M is assigned to.

      ask your sysadmin to configure more swap

      This was already dismissed by the OP, saying 2 Gb is enough.

      Abigail

[reply]

          If you've got to keep intermediary results, consider writing them out to a DBM file or to an RDBMS rather than keeping everything in RAM.

      In case anyone hasn't guessed.. I'm very interested in this thread because I am running into similar problems with some code that I'm writing. I'm not running out of memory but I am noticing that the Perl strucutres I am storing things in are getting very large and unwieldy and RSS when tracked is getting HUGE. Gee... I wonder just how many Perl Monks and non-PM Perl coders are getting sucked into virus remediation right now.

      All that aside: I have thought about using DBMs and/or an RDBMS to store intermediate results for my scripts as well. The problem with that is you tend to lose performance using that approach (especially using DBI) due to the overhead of communicating with the database and the scripts I'm writing are too slow already.

      Of course, as the OP pointed out indirectly the problem with keeping everything in memory is you can concievably run out of memory when your data structures get REALLY huge due to the amount of data you are processing.

          Break your input data set into smaller chunks if possible, process the smaller chunks, then use those results to produce a final agregate result.
      This approach works OK too if your data "cooperates" and lines itself up really nice for you the way you want it. However when you are processing logs you may have related events that you want to track that are many many lines apart in the log files.

      I know I am not coming up with a solution here, but I can feel smellysocks's pain here as I'm trying to solve some of the same problems and I'm not coming up with any good answers either.

      Peter L. Berghold -- Unix Professional
      Peter at Berghold dot Net
      Chat Stuff: AIM:  redcowdawg
      Yahoo: blue_cowdawg
      Cowdawg Philosophy:  Sieze the Cow! Bite the Day!
      Clever Quip:  Nobody expects the Perl Inquisition!
      Non-Perl Passion:   Dog trainer, dog agility exhibitor, brewer of fine Belgian style ales. Happiness is a warm, tired, contented dog curled up at your side and a good Belgian ale in your chalice.
[reply]
Re: Remaining Memory on a system
by Abigail-II (Bishop) on Aug 29, 2003 at 15:50 UTC
    This is something you really want to solve at the OS level, and not at in the program itself. One way to solve this is by setting ulimits, and limit the allowable memory a process can use.

    If you happen to have glance you could also write an advisor script to handle situations like this.

    Abigail

[reply]
Re: Remaining Memory on a system
by aquarium (Curate) on Aug 29, 2003 at 14:16 UTC
    if you must, which NOT the preferred option, then you can use SNMP module to get at the required MIB values. You should do it in a loop, hopefully the loop that actually chews up the memory. Why don't you post up the code that's chewing the memory so badly? Does it have a memory leak?..or could it be re-written?
[reply]

          then you can use SNMP module to get at the required MIB values

      Something tells me that the OP doesn't want to turn on SNMP on a machine that firewall logging is being done on. This IMHO would be a bad thing® and should be avoided.

      Peter L. Berghold -- Unix Professional
      Peter at Berghold dot Net
      Chat Stuff: AIM:  redcowdawg
      Yahoo: blue_cowdawg
      Cowdawg Philosophy:  Sieze the Cow! Bite the Day!
      Clever Quip:  Nobody expects the Perl Inquisition!
      Non-Perl Passion:   Dog trainer, dog agility exhibitor, brewer of fine Belgian style ales. Happiness is a warm, tired, contented dog curled up at your side and a good Belgian ale in your chalice.
[reply]

        First blue_cowdawg, let me say that you are someone I really respect. However, would you consider using a readmore tag on your sig? It's not that it isn't beautiful and very informative because it is. However, attaching a cliffs notes version of your homenode to every post isn't something I'd suggest. Also, the green doesn't work with the css definitions some of your fellow monks use.

        Owner of an "I ♥ Perl" thong,
        Anonymous Monk

[reply]
Re: Remaining Memory on a system
by sgifford (Prior) on Aug 29, 2003 at 15:28 UTC
[reply]
[d/l]
[select]
      I am not concerned about the server running out of memory therfor ulimit will not help.

      What I am concerned about is getting information that my app spent over 8 hours processing and prevent reports like this:

      # more runfwrpt-20030829-0230
      Out of memory during request for 24 bytes, total sbrk() is 317718640 bytes!
      Out of memory during request for 1016 bytes, total sbrk() is 317718640 bytes!
      Out of memory during request for 1016 bytes, total sbrk() is 317718640 bytes!


      The above isn't a very usefull report...

      Even if it only processed half of the files I'd rather have it spit out an error message, and report what it has processed so far.
[reply]
        See the entry for $^M in perlvar(1):
        By default, running out of memory is an untrappable, fatal error. However, if suitably built, Perl can use the contents of "$^M" as an emergency memory pool after die()ing. Suppose that your Perl were compiled with -DPERL_EMERGENCY_SBRK and used Perl's malloc. Then $^M = 'a' x (1 << 16); would allocate a 64K buffer for use in an emergency. See the INSTALL file in the Perl distribution for information on how to enable this option. To discourage casual use of this advanced feature, there is no English long name for this variable.
[reply]
[d/l]

Back to Seekers of Perl Wisdom