Memory Management in Perl

tyric has asked for the wisdom of the Perl Monks concerning the following question:

I'm working on a perpetual program. It runs indefinitly, checking on various servers using ping or SNMP queries. It works like I want it to but it has one major problem. It continualy grabs and uses more ram and process time with each iteration, growing until it become a detriment to the system it's running on.

I have a secondary script, an outside file, that I open and eval. Would this be bloating the memory? This file is simply a hash of data that may change while the script is running. I've set the program up, so that the data can be changed while the script is running.

open(DATA,"data.pl");
   while (<DATA>) {$data .= $_;}
close DATA;

eval $data;
[download]

Here's an example of the data it's calling.

@array = (
        { label => "backup.hostname.com",
        status => "on",
        type => "ping",
        data => "hostname.com",
        user => user@hostname.com,
        error => 'Failed',
        },
);
[download]

But i digress, what I'm looking for is some tutorials on memory management. Where can I find this information, and how can I trim a bloated program while it is running. Similar to undef, but will actually free the memory up for the system.

tyric

Comment on Memory Management in Perl Select or Download Code

Replies are listed 'Best First'.
Re: Memory Management in Perl by dws (Chancellor) on Feb 16, 2002 at 04:25 UTC
I have a secondary script, an outside file, that I open and eval. Would this be bloating the memory? There are reports (see this node and this node) that claim that `eval` leaks memory in older verions of Perl. What version are you running?	[reply] [d/l]
Re: Memory Management in Perl by dash2 (Hermit) on Feb 16, 2002 at 11:09 UTC
First of all 'eval'ing an opened file is pretty risky. (Are you sure nobody else can write `rm -rf /` into that file?) If you want data, why not `use Storable;` to freeze and thaw your data? (Check out Object Serialization Basics for more options.) Second, if you do want to actually run the code in the file, just `do 'data.pl'` or indeed `require 'data.pl'`. As for memory management, check out Devel::Leak on CPAN... I've never used it but it may help. Probably others can think of a few more. dave hj~	[reply] [d/l] [select]
Re: Memory Management in Perl by steves (Curate) on Feb 16, 2002 at 04:49 UTC
I assume you're not continually adding to $data ... That's way too obvious. Besides the note already mentioned about eval leaks in older versions of Perl, the other big one to watch out for is self-referential data; i.e., references that end up pointing back to themselves. That's usually not obvious, as in a directed graph type data structure where a node may end up pointing back to one of its ancestors. I had one of those that blew a Perl program of mine up in a major way. Once identified you can use WeakRef to fix them.	[reply]
Re: Memory Management in Perl by dash2 (Hermit) on Feb 16, 2002 at 11:11 UTC
Another point: if you are pinging people, and not too many, do you actually need to keep running perpetually? Could you just run the thing every 5 minutes on a crontab? This might be simpler than delving into memory management at this stage in the game... dave hj~	[reply]
Re: Re: Memory Management in Perl by tyric (Novice) on Feb 21, 2002 at 02:56 UTC
This is actually an option that we are seriously considering. more on that later. Tyric	[reply]
Re: Memory Management in Perl by thor (Priest) on Feb 16, 2002 at 13:21 UTC
I don't understand why you are storing your data as perl in another file. As an option, you could use a delimited file. Ponder this: `backup.hostname.com\|\|on\|\|ping\|\|hostname.com\|\|user@hostname.com\|\|Failed` [download] You can read this with `split` This way, you don't have to monkey with `eval`. Also, you have a lot of redundancy in your records. I count the word hostname 3 times, plus you have two fields that could probably be represented with a "1" or a "0" (status and error). thor	[reply] [d/l] [select]
Quickie memory leak fix by kjherron (Pilgrim) on Feb 16, 2002 at 16:52 UTC
If you want to fix this without spending a lot of time on it, the simplest thing is probably to perform your tests from a child process. Every time your script starts another series of tests, instead of just diving in to them, start by calling fork(). The child process should perform the tests and exit; the parent process should just call wait() to collect the child process. If the testing process leaks memory, file handles, etc. then they'll be cleaned up when the child process exits. This isn't the elegant, professional solution, but it will solve the problem without requiring a major rewrite of the script.	[reply]