Memory usage of a "sub" in mod

diego_de_lima has asked for the wisdom of the Perl Monks concerning the following question:

Hello fellow monks!

Now that my mod_perl processes are gowing to more than 60mb each, I´m trying to clean up averything I can. Well, it used to be 100mb each, and by eliminating some unused modules I could reduce ir drastically.

But now I´m stucked in the problem of the process memory usage growing big during its lifetime. I have no public variables, and everything is OO. There is some packages which recieves big HTML strings which are latter printed to the browser. Sometimes I can have a 1mb HTML output or even more.

This packages are destroyed by the end of the session, wich is handled by a mod_perl handler (not a cgi file).

I´m doing something like:

sub handler {
 my $r = shift;
 my $page = new Html::page;
 $page->{html} = '...and here it goes...';
 $page->end(); # Does some final processing
 print "Content-type:...".
       $page->{html};
 undef $page;
}

This "end" sub is very simple, and looks like:

sub end {
 my $class = shift;
 $class->{html} = "Header" . $class->{html} . "Footer";
}

Apache2::Satus says it´s consumes about 27kb before serving a request. But, after being executed, it grows to the size of the HTML it processed (109kb in the example below).

BEFORE serving a request I get:

Memory Usage for package Html::page
Totals: 68703 bytes, 67.09 Kb, 0.07 Mb | 1284 OPs
end                 27748 bytes | 573 OPs
new                  7045 bytes | 146 OPs

And AFTER serving a request I get:

Memory Usage for package Html::page
Totals: 151779 bytes, 148.22 Kb, 0.15 Mb | 1284 OPs
end                 109244 bytes | 573 OPs
new                   7045 bytes | 146 OPs

Obviously the mod_perl process grows in the same rate, and it never releases memory. So I end up sometimes with process that grows from 62mb to 120mb in just one request.

Am I doing something wrong? Acording to my knowledge, Perl should release the memory used in the $page->{html} variable just after I undef $page or simply do $page->{html} = ''; Or maybe I´m wrong?

By the way: Linux Fedora 6 with Perl 5.8.8, Apache 2.2.3, mod_perl 2.0.2.

Thanks a lot for your attention monks!

Deigo de Lima

Comment on Memory usage of a "sub" in mod_perl

Replies are listed 'Best First'.
Re: Memory usage of a "sub" in mod_perl by almut (Canon) on May 23, 2008 at 13:14 UTC
You could try Devel::Cycle (or Test::Memory::Cycle, or Devel::Monitor) to check if your `Html::page` object maybe has circular references, which would stop the memory from being freed... In that case, modules like Object::Destroyer might help to tackle the problem (unless you can get rid of the circular refs yourself otherwise). Update: Another thing to be aware of is that even without a true memory leak (due to circular refs), there is still the issue that memory once allocated by the Perl interpreter is usually not being returned to the OS before the interpreter process terminates — though this might be somewhat OS dependent (i.e. I've heard rumours that on Windows, memory actually may be returned to the OS — not sure though). The net effect is that - although memory is being freed and recycled Perl-internally - if you allocate a huge data structure (even if only once), the process size will stay at the maximum memory usage that ever occurred in the lifetime of the process... I think there is a way around that problem by compiling Perl with the configuration settings to not use its own memory pool, but rather have it directly use the (slower) system malloc routines — In other words, if all else fails (and you're absolutely sure you don't have a classical memory leak), this might be worth a try...	[reply] [d/l]
Re: Momory usage of a "sub" in mod_perl by moritz (Cardinal) on May 23, 2008 at 13:14 UTC
Perhaps you have circular references somewhere that prevent structures from being garbage collected. Try Devel::Leek and Devel::Cycle to find them. Once you found them you can weak some of the references with Scalar::Util::weaken.	[reply]
Re: Memory usage of a "sub" in mod_perl by locked_user sundialsvc4 (Abbot) on May 23, 2008 at 13:32 UTC
The operative question is: does the memory consumption keep growing? If it does, then you probably have a circular-loop or something similar that is preventing garbage collection. But remember... a computer is lazy. (Lazier, in fact, than a good Perl programmer, if such a thing is possible...) It's not going to reduce the working-set size of a process unless there is actual competition for the storage-resource. There's no value, after all, in reclaiming a virtual-memory page that's going to incur a page-fault a few milliseconds later...
Re: Memory usage of a "sub" in mod_perl by perrin (Chancellor) on May 23, 2008 at 17:07 UTC
It is normal for a process to remain at the maximum size it has reached. Because you undef $page, that memory should be available to perl for other uses. If you see the process continue to grow, that's when you need to worry. I don't understand how 109K of HTML causes your process to grow 58MB. Do you actually have strings that take up that much space in a single request?	[reply]
Re^2: Memory usage of a "sub" in mod_perl by diego_de_lima (Beadle) on May 23, 2008 at 19:33 UTC
Thnaks for all answers, they were quite usefull. I´ve many circular refs, and now I weakened them, which reduced slightly the memory usage. But still the process grows and don´t free its memory. 100K HTML isn´t a big problem, but some times I have a 10mb ou 30mb structure in memory, which then generates a 3 mb HTML. And thats the problem. I´m sure there aren´t circular refs anymore, but system doesn´t reclaims memory. I think this is the normal behavior, and Apache2::SizeLimit is the only solution for this. Am I right? But I still can´t undestand why it says the sub "end" is eating all this memory! I would understand if it says there is a global variable or something, but why a sub? For example, the perl-stats Memory Usage of the Html::page package says: 128: undef PV 44 bytes html 129: undef NULL 24 bytes 0xa7f4cb8 130: undef PV 37740 bytes Content-type: text/html Cache-Control: no-cache,no-store,max-age=0 Pragma: no-cache <html>... ALL MY PAGE HERE 2 MB OF DATA!!! Why is it still arround? I´ve done a $page->{html} = ''; in the DESTROY method and I´m sure it was invoked. And, by the way, it has nothing to do with the sub "end". "end" just parses $page->{html}, then flushes it to the browse and then undef $page->{html}. Shouldn´t "undef" be enouth? Thanks monks! Diego de Lima	[reply]
Re^3: Memory usage of a "sub" in mod_perl by shmem (Chancellor) on May 23, 2008 at 19:53 UTC
and then undef $page->{html}. Shouldn't "undef" be enough? That depends upon what that hash entry actually is. If it is a reference (or something containing a reference) to data which is held elsewhere, you just clear that slot from the reference, but not the referenced data. Consider: `perl -le '$foo = "bar"; $h->{foo} = \$foo; undef $h->{foo}; print "foo + :$foo:"' foo :bar:` [download] which is different to `perl -le '$foo = "bar"; $h->{foo} = \$foo; undef ${$h->{foo}}; print " +foo :$foo:"' foo ::` [download] where the referenced data is cleared. --shmem _($_=" "x(1<<5)."?\n".q·/)Oo. G°\ / /\_¯/(q / ---------------------------- \__(m.====·.(_("always off the crowd"))."· ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}	[reply] [d/l] [select]
Re^4: Memory usage of a "sub" in mod_perl by diego_de_lima (Beadle) on May 23, 2008 at 20:09 UTC
Re^5: Memory usage of a "sub" in mod_perl by shmem (Chancellor) on May 23, 2008 at 20:26 UTC
Re^3: Memory usage of a "sub" in mod_perl by perrin (Chancellor) on May 23, 2008 at 20:08 UTC
If you run the exact same request multiple times and the process size grows every time, you have a leak, probably a circular reference. If it stops growing but remains at the larger size, that is completely normal and the only way to avoid it is to structure your code so that it doesn't need to have all the data in memory at once.	[reply]
Re: Memory usage of a "sub" in mod_perl by Proclus (Beadle) on May 29, 2008 at 09:07 UTC
Would you use a C based HTML template library like Clearsilver to play with your HTML files/templates? I'm currently working on a mod_perl/CGI::App/Clearsilver project, so far everything looks good. I don't have MB HTML files though. ------------------------------ "geneva will not marry you no matter how much you love Larry Wall." Geneva Wall	[reply]