Find largest variables

dinomite has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Find largest variables by BrowserUk (Patriarch) on Dec 19, 2008 at 23:22 UTC
Take a look at Devel::DumpSizes which will use Devel::Size dump the size of everything it can find at a given point in the program. And I recommend (I would, I modified it) the latest, but unauthorised version of Devel::Size. It is far quicker; uses far less memory for its internal accounting; and is far more reliable (less likely to trap) than the authorised version. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]
Re: Find largest variables by chromatic (Archbishop) on Dec 19, 2008 at 23:05 UTC
Assuming you have all of your variables properly scoped (that is, you're using lexicals declared with my), the most likely culprit is processing a large list of information, such as lines from a file, in a memory-inefficient way. Devel::Size can help you profile variables, but my guess is that you're processing a large file somehow. We can't be more specific without seeing some code however.	[reply]
Re: Find largest variables by dinomite (Sexton) on Dec 20, 2008 at 01:21 UTC
Thanks folks, I'm going to give Devel::DumpSizes a try. This is a long-running process (mod_perl handler) that is quite large (10k lines) which I inherited, so trying to divine the problem by perusing the code is cumbersome at best. It eats memory all day long, requiring that Apache be restarted every day or so.	[reply]
Re^2: Find largest variables by BrowserUk (Patriarch) on Dec 20, 2008 at 08:43 UTC
I'd suggest that you add a few lines just before the mod script "exits". I'm aware that mod_perl script don't exit, but I mean at a convenient point after the html request that caused it to be invoked has completed (to avoid impacting your users). I can't be more precise about it as I know naff all about mod_perl. Something like this: `... use Devel::Size qw[ total_size ]; ... unless( int( time ) % 3600 ) { open my $sizelog, '>', '/some/path/' . localtime() . '.log' or war +n $!; print $sizelog "$_ =>", total_size( $::{ $_ } ) for sort keys %::; close $sizelog; }` [download] The idea is tha once per hour (assumes that you have a fairly steady flow of activeity on the site), that you dump the total sizes of every package known, to a time-named file. You'll get output that looks something like this: :> 378 ☼ :> 418 ↕ :> 262 ▬ :> 345 ↨ARNING_BITS :> 400 ↑ :> 330 ! :> 403 " :> 292 $ :> 272 + :> 552 - :> 552 / :> 382 0 :> 386 1 :> 425 2 :> 407 <none>:: :> 460 @ :> 950 ARGV :> 390 Devel::Size: Calculated sizes for compiled regexes are incompatible, a +nd probably always will be AutoLoader:: :> 25020 BEGIN :> 272 CORE:: :> 930 Carp:: :> 17283 Devel::Size: Calculated sizes for compiled regexes are incompatible, a +nd probably always will be Config:: :> 91676 DB:: :> 850 Devel::Size: Calculated sizes for compiled regexes are incompatible, a +nd probably always will be Devel:: :> 22891 Devel::Size: Calculated sizes for compiled regexes are incompatible, a +nd probably always will be DynaLoader:: :> 55547 ENV :> 6491 Devel::Size: Calculated sizes for compiled regexes are incompatible, a +nd probably always will be Exporter:: :> 23537 INC :> 1710 IO:: :> 928 Internals:: :> 3421 Mac:: :> 1725 PerlIO:: :> 2363 Regexp:: :> 933 SIG :> 3595 STDERR :> 274 STDIN :> 272 STDOUT :> 274 UNIVERSAL:: :> 1857 VMS:: :> 1248 Win32:: :> 11242 ] :> 355 _ :> 345 _<..\perlio.c :> 336 _<..\universal.c :> 348 _<..\xsutils.c :> 340 _<.\win32.c :> 328 _<Size.c :> 339 _<c:/Perl/site/lib/auto/Devel/Size/Size.dll :> 479 _<dl_win32.c :> 330 _<perllib.c :> 328 attributes:: :> 947 Devel::Size: Calculated sizes for compiled regexes are incompatible, a +nd probably always will be main:: :> 299655 stderr :> 274 stdin :> 272 stdout :> 274 strict:: :> 6427 total_size :> 844 utf8:: :> 4344 Devel::Size: Calculated sizes for compiled regexes are incompatible, a +nd probably always will be vars:: :> 9566 warnings:: :> 58863 [download] With luck that will prevent you from being overwhelmed with data (a smallish, easily compared file, once per hour), that should show you which package(s) are accumulating memory over time. Once you have that information, you can then concentrate your efforts there. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l] [select]
Re: Find largest variables by eye (Chaplain) on Dec 20, 2008 at 06:31 UTC
If variable profiling does not solve the problem, you might check for the use of faulty recursion in the program. Once execution recurses, variables in the calling context can be trapped until execution returns to the calling context. This can occasionally occur inadvertently when a collection of functions/methods call each other in complex ways.	[reply]