venki has asked for the wisdom of the Perl Monks concerning the following question:

I have written a perl progam using forks. This runs as a process in unix environment. The program runs fine without any errors and does whatever it is supposed to do. Also this process uses very low memory when startup but as time passes the memory used by the program increases dramatically. May any one give me general tips what to look for when this happens. what is the best way to overcome this issue. Thanks

Replies are listed 'Best First'.
Re: Perl program process
by Zaxo (Archbishop) on Jul 01, 2002 at 22:12 UTC

    You don't provide enough information for anything but guesses.

    Some places to look:

    1. Data structures that grow but are never pruned, a program design error<./li>
    2. Reading endless data, like looking for a newline in /dev/zero. Unlikely; I just mention it because I did that myself yesterday. :)
    3. Circular references. Garbage collection does not pick these up because the refcount never goes to zero.
    4. Closures. Their code is not subject to garbage collection, if I understand correctly.
    5. Leaving zombies. If the long-running code does not wait or respond to SIGCHLD properly, the os cannot release an exited child's process data. Probably not a significant amount of memory, you'd run into the process limit first.

    The system resource monitor, 'top', will inform you which process has the memory hog. The 'ps' utility will show Z in the status column for zombies. A -DDEBUGGING Perl will help spot the culprit.

    After Compline,
    Zaxo

Re: Perl program process
by hacker (Priest) on Jul 01, 2002 at 22:01 UTC
    Depending on the nature of your code (monolithic, subs, OO), you may find use in using Devel::Dprof to trace the execution time of your code and from there find out where you can optimize. You didn't say whether or not your program was a CGI, or standalone. Can you verify that the memory consumption issues are inside the perl, and not outside, such as calling external executables which leak memory and don't free() it back up?

    Another one to try is Devel::Peek has a 'memory footprint debugging' API also. I haven't used either of these, but they look promising towards determining your issue of memory consumption.

Re: Perl program process
by ehdonhon (Curate) on Jul 01, 2002 at 22:08 UTC

    The reason for this is not in the way that Perl is forking, but the way unix itself handles forking. I strongly recommend you get a set of the "Unix Network Programming" books by W. Richard Stevens if you do not already have them and plan on doing a lot of work with forking programs.

    The reason that this memory problem is happening is that every time you fork, the two new processes are sharing the same memory pages, but they are sharing them on a copy-on-write fashion. So, the moment one of the two processes tries to update any data stored within a memory page, both processes end up getting their own copy of the entire page. This is a very efficient way of doing things when your children live for a short while and don't do much writing. However, the longer your children live (or the more your children need to write to memory), the less efficient this becomes.

    If you have children that need to live for a very long time, you should evaluate your situation and decide wether you are better off in a forking environment, or if you would be better off working with threads.

Re: Perl program process
by bronto (Priest) on Jul 02, 2002 at 08:26 UTC

    I find very difficult to help you with the few information you gave. So I'll give you a general advice

    When I need to use some memory temporarily (say I need temporary variables or data structures or objects...), I usually isolate temporary items as lexical variables inside a block. That way, when the execution of the block comes to end, lexicals disappear -unless you created a closure or circular references or similar things...

    For example, say I want to encrypt a password using crypt. The user writes the password on STDIN and I need to do some basic checks before encriptying. If at the end of the process I want to retain just the encrypted password I'd do this way:

    my $crypted ; { my $checker = My::Password::Checker->new() ; my $password = <STDIN> ; chomp $password ; die "too short!" unless length $password < 4 ; die "too obvious" if $checker->check_obvious($password) ; # other tests go here... my $salt = generate_random_salt() ; $crypted = crypt($password,$salt) ; } # $password, $salt, $checker and any other variable # declared by my inside the block disappear here :-) # $crypted was declared outside the block, so I can... do_whatever_with($crypted) ;

    I hope this helps

    Ciao!
    --bronto

    # Another Perl edition of a song:
    # The End, by The Beatles
    END {
      $you->take($love) eq $you->made($love) ;
    }

Re: Perl program process
by Abigail-II (Bishop) on Jul 02, 2002 at 09:21 UTC
    Your description is too vague to be useful. If the memory increases dramatically, it's most likely caused by the program using lots of datastructures.

    Best tip to where to look into: your program source. Analyze the flow, try to figure out what's happening.

    Abigail