in reply to Extensive memory usage
Using my to tell Perl when you don't need a variable anymore is a good start, but Perl uses reference counting for memory management, so you might run into circular references like the following:
my $bar = {}; my $foo = { bar => \$bar }; $bar->{foo} = \$foo;
Here, $bar points to $foo and reverse, so there is no way that Perl will ever release the two variables back until the process exits.
There are a number of solutions to circumvent the problem. The easiest way is to split up your program into several programs, which all process a small amount of the data and then exit, thus cleaning up the memory.
There is also weaken in Scalar::Util, which creates a weak reference, which will not prevent the target from being collected. To know which references to weaken is not easy though.
You can also try to analyze your code and find out where circular references are, and manually break them, by setting in the example $foo->{bar} = undef.
A second thing might be that your program is simply creating too complex memory structures - for example, I can well imagine an 10 MB XML file eating up lots of memory - so you might want to consider changing your data structures or changing from a DOM XML parser to an event based parser. Such restructuring is not easy though.
|
|---|