Using my to tell Perl when you don't need a variable anymore is a good start, but Perl uses reference counting for memory management, so you might run into circular references like the following:

my $bar = {};
my $foo = { bar => \$bar };
$bar->{foo} = \$foo;

[download]

Here, $bar points to $foo and reverse, so there is no way that Perl will ever release the two variables back until the process exits.

There are a number of solutions to circumvent the problem. The easiest way is to split up your program into several programs, which all process a small amount of the data and then exit, thus cleaning up the memory.

There is also weaken in Scalar::Util, which creates a weak reference, which will not prevent the target from being collected. To know which references to weaken is not easy though.

You can also try to analyze your code and find out where circular references are, and manually break them, by setting in the example $foo->{bar} = undef.

A second thing might be that your program is simply creating too complex memory structures - for example, I can well imagine an 10 MB XML file eating up lots of memory - so you might want to consider changing your data structures or changing from a DOM XML parser to an event based parser. Such restructuring is not easy though.

In reply to Re: Extensive memory usage by Corion
in thread Extensive memory usage by TheMarty

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.