Re^2: How to put a fat program on a (memory) weight-loss diet?

I guess that's the first question. I did an strace on it, and the first thing that I see is that the mmap() calls total up to 12,512,290. (But this includes 7,340,032 that did not succeed...)

“Ahem... let me try that again... awk, if you please... thank you. Now, where was I?”

The mmap calls for the program as it runs to normal completion total 8,329,178 bytes. Which, frankly, doesn't seem like a lot.

I really don't know what else might be found in an strace output that would point the way to more memory-allocations. I do know a lot about the program itself.

This is a Moose / Mojolicious program which uses DBIx::Class for database-access and Template::Toolkit. It doesn't get fancy with a lot of memory allocation... at least, not in my code. All in all, I think it's a fairly “run of the mill” application that does use a lot of CPAN material.

Since the program does run, after a fashion, even with these constraints, about the only logical culprit I can think of is DBIx::Class. That is to say, “that's what tips it over the edge of the cliff,” whereupon it bounces and goes on to successfully produce the template and so-on.

But what I am hoping to find is something “quick 'n dirty” that will substantially reduce the footprint, and do so quickly without demanding major logic-changes. (Y’know, like “rewriting it in PHP...” ;-) )

Edit: There are also brk() calls, the first one setting the limit to 0x814df20 and the last to 0xa267000, a span of 34mb... which does not look good. :~}

Learning... it's not for sissies...

Replies are listed 'Best First'.
Re^3: How to put a fat program on a (memory) weight-loss diet? by BrowserUk (Patriarch) on Mar 04, 2009 at 14:38 UTC
You might be able to gather more information on what is using the memory using Devel::Size v0.72. If you run: `use Devel::Size qw[ total_size ]; ... printf "%s :: %.f\n", $_, total_size( $::{ $_ } ) for keys %::;;` [download] At some point before the crash occurs (experiment), then it will produce a list something like: / :: 382 stderr :: 289 SIG :: 3705 , :: 399 utf8:: :: 4359 " :: 347 _<c:/Perl/site/lib/auto/Devel/Size/Size.dll :: 479 DynaLoader:: :: 55772 Devel:: :: 22977 strict:: :: 6435 stdout :: 289 AllocMemory :: 2263 ↕ :: 262 \| :: 383 _<c:/Perl/lib/auto/Time/HiRes/HiRes.dll :: 463 Mac:: :: 1706 CleanString :: 15703 Regexp:: :: 948 _code :: 443 UNIVERSAL:: :: 1872 overload:: :: 39770 $ :: 287 time :: 829 NewString :: 13443 size :: 820 Data:: :: 111665 - :: 679 _<..\universal.c :: 363 _<HiRes.c :: 343 BEGIN :: 287 ! :: 380 IO:: :: 943 ☼ :: 399 total_size :: 844 ↑ :: 345 pp :: 12454 _ :: 345 _<c:/Perl/site/lib/auto/Win32/API/API.dll :: 471 + :: 679 Exporter:: :: 81873 Internals:: :: 3436 STDIN :: 287 Config:: :: 91676 warnings:: :: 59111 DB:: :: 850 Time:: :: 33192 _<.\win32.c :: 343 ▬ :: 345 _<perllib.c :: 343 2 :: 388 _<API.c :: 335 cmpthese :: 27607 1 :: 406 ↨ARNING_BITS :: 400 CORE:: :: 930 _<Size.c :: 339 attributes:: :: 962 stdin :: 287 ARGV :: 405 INC :: 2857 Scalar:: :: 1789 ENV :: 6689 ? :: 395 vars:: :: 9565 subs:: :: 3764 _<..\perlio.c :: 351 main:: :: 720485 AutoLoader:: :: 24976 Carp:: :: 31355 VMS:: :: 1229 Win32:: :: 256452 PerlIO:: :: 2378 0 :: 431 :: 562 _<..\xsutils.c :: 355 @ :: 950 ApiLink :: 10523 Benchmark:: :: 131181 STDOUT :: 289 ] :: 355 3 :: 386 ↨ :: 383 MIME:: :: 1215 STDERR :: 289 _<dl_win32.c :: 330 <none>:: :: 460 sleep :: 839 [download] Which is a crude assessment of the memory being used by each of the packages you have loaded (plus other stuff). That may allow you to zero in on one particular package that is being profligate and then look at that in more detail by examining its symbol table more closely: [0] Perl> printf "%s :: %.f\n", $_, total_size( $::{ $_ } ) for keys % +{ Benchmark:: };; timesum :: 112 __ANON__ :: 112 cpu_a :: 112 a :: 112 n_to_for :: 112 _doeval :: 112 iters :: 112 _Usage :: 112 Min_CPU :: 112 init :: 112 cmpthese :: 27607 debug :: 112 export :: 112 new :: 112 timestr :: 112 EXPORT_TAGS :: 112 timethis :: 112 countit :: 112 EXPORT_OK :: 112 confess :: 112 EXPORT_FAIL :: 112 timediff :: 112 cpu_c :: 112 b :: 112 ISA :: 112 export_to_level :: 112 Default_Style :: 112 timethese :: 112 time :: 829 Cache :: 112 cpu_p :: 112 Debug :: 112 clearcache :: 112 BEGIN :: 287 Do_Cache :: 112 runloop :: 112 timedebug :: 112 real :: 112 usage :: 112 clearallcache :: 112 timeit :: 112 EXPORT :: 112 croak :: 112 import :: 112 disablecache :: 112 enablecache :: 112 Min_Count :: 112 carp :: 112 Default_Format :: 112 VERSION :: 112 mytime :: 112 [download] Make sure that you get my unofficial v0.72, as previous versions use a substantial amount of memory for internal tracking that skews the results and would probably push you over the limits. My version doesn't suffer that defect. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l] [select]
Re^4: How to put a fat program on a (memory) weight-loss diet? by locked_user sundialsvc4 (Abbot) on Mar 04, 2009 at 17:46 UTC
I don't get the expected results in the second case. Although the first output gives a size of `5113317` for my main application namespace, an individual listing of it shows only a handful of names with a size of `112` attached to each one. Edit: Okay, I worked it a little-bit more and found out both “what `%::` is for” and that it is a hash-of-hashes. (Sounds dumb, yes, but new-to-me.) And basically what it seems to come down to is ... ummm ... “gee, it all adds up!” More than 4 megabytes of now-known-to-be precious memory is occupied by a `DBIx::Class::Schema`! And nearly 30 megabytes in all of ... “stuff.” Uh, oh... I don't think this is gonna be a situation where I can just wrap a magic weight-loss belt around it.
Re^5: How to put a fat program on a (memory) weight-loss diet? by BrowserUk (Patriarch) on Mar 04, 2009 at 18:34 UTC
Sorry. I typoed. Try substituting "Your application namespace" for "Benchmark" in the following: `printf "%s :: %.f\n", $_, total_size( $Benchmark::{ $_ } ) for keys %{ Benchmark:: };;` [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^3: How to put a fat program on a (memory) weight-loss diet? by stvn (Monsignor) on Mar 04, 2009 at 22:05 UTC
This is a Moose / Mojolicious program which uses DBIx::Class Well, that sounds like your problem right there. You also mention in the original post that it is a CGI program, which Moose have been known to attack and bite ;) First, if it really is a vanilla CGI program, try switching to a persistent environment this will use less memory over time since a lot of Moose and DBIx::Class only grab memory once at compile time and are they pretty reasonable thereafter. If that doesn't work, or if you already are in a persistent environment. Try swithing Moose to Mouse (or possibly Any::Moose) as that will at least reduce that part of the memory footprint. -stvn	[reply]