Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

way to find module memory usage?

by perl-diddler (Chaplain)
on Jun 06, 2018 at 20:05 UTC ( #1216051=perlquestion: print w/replies, xml ) Need Help??

perl-diddler has asked for the wisdom of the Perl Monks concerning the following question:

I added some testing routines in a prog I have, and added code to disable them unless I select the tests from the command line.
if (DEBUG_ENABLED) { ... 54 lines of somewhat dense code (1.7k chars). }

What I want to find out is if disabling it really reduces memory size. FWIW, DEBUG_ENABLED is optimized to a constant, so theoretically, perl should be able to throw away anything in the braces and I should see a smaller package size. Am trying to find out if theory matches reality. :-)

The memory routines I've seen don't seem to allow inspecting the size of a package, especially w/r/t breaking down:
code 47.5KB
data 30KB

Of course ideally it would be something like this FF extension (showing memory related to this page):
(about:memory shows the usage for the entire browser, and the detail view shows that sources are stored where in memory).

│ ├──19.49 MB (04.40%) -- top(http://perlmonks.org/?node=Seekers of Perl Wisdom, id=9)
│ │ ├──12.25 MB (02.77%) -- cached
│ │ │ ├───6.31 MB (01.42%) -- window(http://perlmonks.org/?node_id=6364;user=438862)
│ │ │ │ ├──5.44 MB (01.23%) -- layout
│ │ │ │ │ ├──4.94 MB (01.12%) ── style-sets
│ │ │ │ │ └──0.51 MB (00.11%) ++ (6 tiny)
│ │ │ │ └──0.86 MB (00.20%) ++ (4 tiny)
│ │ │ └───5.95 MB (01.34%) -- window(http://perlmonks.org/?node_id=438862)
│ │ │ ├──5.26 MB (01.19%) -- layout
│ │ │ │ ├──4.94 MB (01.12%) ── style-sets
│ │ │ │ └──0.33 MB (00.07%) ++ (6 tiny)
│ │ │ └──0.68 MB (00.15%) ++ (4 tiny)
│ │ ├───6.71 MB (01.52%) -- active/window(http://perlmonks.org/?node=Seekers of Perl Wisdom)
│ │ │ ├──5.76 MB (01.30%) -- layout
│ │ │ │ ├──4.94 MB (01.12%) ── style-sets
│ │ │ │ └──0.82 MB (00.19%) ++ (6 tiny)
│ │ │ └──0.95 MB (00.22%) ++ (4 tiny)
│ │ └───0.52 MB (00.12%) ++ js-zone(0x1c187000)
│ ├──11.51 MB (02.60%) -- top(none)
│ │ ├───7.24 MB (01.63%) -- ghost
│ │ │ ├──4.78 MB (01.08%) -- window(https://log.perl.org/2018/05/goodbye-search-dot-cpan-dot-org.html)
│ │ │ │ ├──4.78 MB (01.08%) ++ js-compartment(https://log.perl.org/2018/05/goodbye-search-dot-cpan-dot-org.html)
│ │ │ │ └──0.00 MB (00.00%) ++ dom
│ │ │ └──2.46 MB (00.56%) ++ window(https://log.perl.org/2018/03/goodbye-bitcard.html#comment-form)
│ │ └───4.27 MB (00.96%) ++ detached/window(system)

Anyway, just the code amount is really what I'm interested in, since there aren't any permanent vars in that section, the other vars should just be on the stack.

Seems like there are several related to showing data sizes, and there fairly good tools for measuring cpu usage, but not so much when it comes to code usage...

Ideals? Suggestions?

Replies are listed 'Best First'.
Re: way to find module memory usage?
by BrowserUk (Patriarch) on Jun 06, 2018 at 20:27 UTC

    It won't give you the sizes directly, but the easiest way to convince yourself that your debug code is being exclused is to use B::Deparse, which for reasons I do not understand is invoked by adding -MO=Deparse to your command line.

    This is what the output looks like when debug is enabled:

    C:\test>type junk.pl #! perl -slw use strict; use constant DEBUG => 1; if( DEBUG ) { print "Debugging..."; } print "Not debugging..."; C:\test>junk.pl Debugging... Not debugging... C:\test>perl -MO=Deparse junk.pl BEGIN { $^W = 1; } BEGIN { $/ = "\n"; $\ = "\n"; } use constant ('DEBUG', 1); use strict 'refs'; do { print 'Debugging...' }; print 'Not debugging...'; junk.pl syntax OK

    Note how the conditional block has been replaced by a do block.

    And this is what it looks like when debug is disabled:

    C:\test>type junk.pl #! perl -slw use strict; use constant DEBUG => 0; if( DEBUG ) { print "Debugging..."; } print "Not debugging..."; C:\test>junk.pl Not debugging... C:\test>perl -MO=Deparse junk.pl BEGIN { $^W = 1; } BEGIN { $/ = "\n"; $\ = "\n"; } use constant ('DEBUG', 0); use strict 'refs'; '???'; print 'Not debugging...'; junk.pl syntax OK

    Note that the entire conditional block has been replaced by '???'.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
    In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit
      That certainly helps, though I would still like to see numbers!

      Think if you did bench mark testing and the only result you got back was whether or not routine "A" was faster than routine "B".

      Certainly helpful, but you'd probably want to know numbers -- by how much...etc.

      Thanks!

        That certainly helps, though I would still like to see numbers!

        My guess as to why noone has written a module for that is that it is of no great concern.

        Example, I have a generated perl file that basically was created to optimise a very heavily nested set of loops (over 100), by unrolling those loops.

        It is just under 4MB in size:

        07/06/2018 01:44 3,959,242 gened.pl 1 File(s) 3,959,242 bytes

        And consists of just over 50kloc:

        C:\test>wc -l gened.pl 50405 gened.pl

        When run, it executes 2.3 million lines:

        C:\test>perl -d:Trace gened.pl >log 2>&1 C:\test>dir log 07/06/2018 01:49 2,334,425 log 1 File(s) 2,334,425 bytes

        I just tacked these two line to the bottom of the file:

        sub mem2{ `tasklist /nh /fi "PID eq $$"` =~ m[(\S+ K)$]; } print mem2;

        And when run, it produces:

        [ 1:52:14.81] C:\test>tail gened.pl X4: L4( $s, $e, $p ); X3: L3( $s, $e, $p ); X2: L2( $s, $e, $p ); X1: L1( $s, $e, $p ); } Xsub( 99, 88, 77 ); sub mem2{ `tasklist /nh /fi "PID eq $$"` =~ m[(\S+ K)$]; } print mem2; [ 1:53:47.89] C:\test>gened 187,124 K

        So 50kloc of code compiles to <200MB of memory.

        Unless you're using Moose, the size of the code usually pales in significance with respect to the data. And the great benefit of properly optimised debug blocks is the removal of the condition test within inner loops and the runtime it saves.

        Not what you want to hear, but it might set your mind at rest about a few lines of debug.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
        In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit
Re: way to find module memory usage?
by BrowserUk (Patriarch) on Jun 07, 2018 at 01:44 UTC

    Hm. Vague memories from when I created a better version of Devel::Size.

    If you print the keys of the default stash %{main::} and filter it for those that keys that end in '::', it gives you a list of all the stashes (codespaces) in your program at that point:

    [0]{} Perl> print for grep /::$/, version:: Tie:: Devel:: Cwd:: Regexp:: UNIVERSAL:: overload:: File:: feature:: ActivePerl:: Config:: warnings:: Time:: EPOC:: Win32CORE:: attributes:: vars:: subs:: XSLoader:: main:: AutoLoader:: Carp:: Win32:: Used:: threads:: MIME:: ActiveState:: utf8:: constant:: re:: mro:: DynaLoader:: strict:: Term:: Sub:: used:: Dos:: Data:: List:: IO:: Exporter:: Internals:: DB:: CORE:: Scalar:: VMS:: PerlIO:: Benchmark:: c::

    If you use Devel::Size total_size() against each of those stashes, it gives you a number that (more or less) represents the size of those stashes:

    [0]{} Perl> print "$_: ", total_size( \%{$_} ) for grep /::$/, keys %{ +main::};; version::: 8400 Tie::: 4339 Devel::: 25803 Cwd::: 118168 Regexp::: 1599 UNIVERSAL::: 1967 overload::: 54595 File::: 80032 feature::: 20416 ActivePerl::: 139164 Config::: 84324 warnings::: 84493 Time::: 39560 EPOC::: 481 Win32CORE::: 657 attributes::: 658 vars::: 14306 subs::: 4786 XSLoader::: 26283 main::: 1194552 AutoLoader::: 38435 Carp::: 39827 Win32::: 9600 Used::: 1872 threads::: 52381 MIME::: 914 ActiveState::: 74236 utf8::: 3147 constant::: 28102 re::: 40244 mro::: 3242 DynaLoader::: 59460 strict::: 8389 Term::: 685 Sub::: 1784 used::: 490 Dos::: 480 Data::: 163029 List::: 20470 IO::: 685 Exporter::: 119426 Internals::: 2469 DB::: 541 CORE::: 672 Scalar::: 38034 VMS::: 1579 PerlIO::: 1833 Benchmark::: 197989 c::: 203

    You can take that one stage further, and iterate the keys of those stashes and you get a list of subs (and other assorted entities) which you can run total_size() against.

    (The output shown is a partial list of the stuff loaded into my REPL environment; the full thing runs to well over 1000 lines):

    for my $stash ( grep /::$/, keys %{main::} ) { print "$stash ", total_size \%{$stash}; for my $func ( keys %{$stash} ){ print "${stash}${func}:", total_size( $func ); } };; version:: 8400 version::normal: 80 version::vcmp: 78 version::parse: 79 version::boolean: 81 version::stringify: 83 version::AUTOLOAD: 82 version::qv: 76 version::("": 77 version::(<=>: 78 version::new: 77 version::(cmp: 78 version::(0+: 77 version::DESTROY: 81 version::noop: 78 version::(): 76 version::declare: 81 version::(nomethod: 83 version::(bool: 79 version::is_qv: 79 version::numify: 80 version::is_alpha: 82 Tie:: 4339 Tie::Hash::: 80 Devel:: 25803 Devel::Size::: 80 Cwd:: 118168 Cwd::__ANON__: 82 Cwd::sys_abspath: 85 Cwd::_epoc_cwd: 83 Cwd::sys_cwd: 81 Cwd::_perl_abs_path: 88 Cwd::cwd: 77 Cwd::_dos_cwd: 82 Cwd::_vms_abs_path: 87 Cwd::_backtick_pwd: 87 Cwd::_carp: 79 Cwd::chdir_init: 84 Cwd::getcwd: 80 Cwd::_win32_cwd: 84 Cwd::fastcwd_: 82 Cwd::EXPORT_OK: 83 Cwd::_NT_cwd: 81 Cwd::_os2_cwd: 82 Cwd::ISA: 77 Cwd::fastcwd: 81 Cwd::_croak: 80 Cwd::REALPATH: 82 Cwd::DIR: 77 Cwd::fast_abs_path: 87 Cwd::_qnx_cwd: 82 Cwd::bootstrap: 83 Cwd::fastgetcwd: 84 Cwd::BEGIN: 79 Cwd::PARENT: 80 Cwd::fast_realpath: 87 Cwd::_vms_cwd: 82 Cwd::_perl_getcwd: 86 Cwd::realpath: 82 Cwd::EXPORT: 80 Cwd::getdcwd: 81 Cwd::chdir: 79 Cwd::VERSION: 81 Cwd::_qnx_abs_path: 87 Cwd::abs_path: 82 Regexp:: 1599 Regexp::DESTROY: 81 UNIVERSAL:: 1967 UNIVERSAL::isa: 77 UNIVERSAL::VERSION: 81 UNIVERSAL::can: 77 UNIVERSAL::DOES: 78 overload:: 54595 overload::ops: 77 overload::mycan: 79 overload::unimport: 82 overload::BEGIN: 79 overload::remove_constant: 89 overload::OVERLOAD: 82 overload::nil: 77 overload::Method: 80 overload::AddrRef: 81 overload::Overloaded: 84 overload::StrVal: 80 overload::constant: 82 overload::OverloadedStringify: 93 overload::package: 81 overload::import: 80 overload::ov_method: 83 overload::constants: 83 overload::VERSION: 81 File:: 80032 File::Spec::: 80 File::Basename::: 84 feature:: 20416 feature::unknown_feature_bundle: 96 feature::unimport: 82 feature::VERSION: 81 feature::croak: 79 feature::import: 80 feature::unknown_feature: 89 ActivePerl:: 139164 ActivePerl::BUILD: 79 ActivePerl::PRODUCT: 81 ActivePerl::CHANGELIST: 84 ActivePerl::_CONFIG_HEAVY: 87 ActivePerl::Config::: 82 ActivePerl::VERSION: 81 Config:: 84324 Config::TIEHASH: 81 Config::config_vars: 85 Config::CLEAR: 79 Config::launcher: 82 Config::Config: 80 Config::NEXTKEY: 81 Config::fetch_string: 86 Config::summary: 81 Config::AUTOLOAD: 82 Config::byteorder: 83 Config::config_sh: 83 Config::BEGIN: 79 Config::EXISTS: 80 Config::myconfig: 82 ... used:: 490 used::VERSION: 81 Dos:: 480 Dos::GetCwd: 80 Data:: 163029 Data::Dump::: 80 List:: 20470 List::Util::: 80 IO:: 685 IO::Handle::: 82 Exporter:: 119426 Exporter::export_fail: 85 Exporter::__ANON__: 82 Exporter::export_tags: 85 Exporter::Cache: 79 Exporter::Heavy::: 81 Exporter::Verbose: 81 Exporter::export_ok_tags: 88 Exporter::Debug: 79 Exporter::export: 80 Exporter::ExportLevel: 85 Exporter::EXPORT: 80 Exporter::import: 80 Exporter::EXPORT_FAIL: 85 Exporter::as_heavy: 82 Exporter::require_version: 89 Exporter::FailCache: 83 Exporter::VERSION: 81 Exporter::export_to_level: 89 Internals:: 2469 Internals::SvREFCNT: 82 Internals::hv_clear_placeholders: 95 Internals::hash_seed: 83 Internals::SvREADONLY: 84 Internals::HvREHASH: 82 Internals::rehash_seed: 85 DB:: 541 DB::args: 78 CORE:: 672 CORE::GLOBAL::: 82 Scalar:: 38034 Scalar::Util::: 80 VMS:: 1579 VMS::Filespec::: 84 PerlIO:: 1833 PerlIO::get_layers: 84 PerlIO::Layer::: 81 Benchmark:: 197989 Benchmark::timesum: 81 Benchmark::__ANON__: 82 Benchmark::cpu_a: 79 Benchmark::a: 75 Benchmark::n_to_for: 82 Benchmark::_doeval: 81 Benchmark::iters: 79 Benchmark::_Usage: 80 Benchmark::Min_CPU: 81 Benchmark::init: 78 Benchmark::cmpthese: 82 Benchmark::debug: 79 Benchmark::export: 80 Benchmark::new: 77 Benchmark::timestr: 81 Benchmark::EXPORT_TAGS: 85 Benchmark::timethis: 82 Benchmark::countit: 81 Benchmark::EXPORT_OK: 83 Benchmark::confess: 81 Benchmark::EXPORT_FAIL: 85 Benchmark::timediff: 82 Benchmark::cpu_c: 79 Benchmark::b: 75 Benchmark::ISA: 77 Benchmark::export_to_level: 89 Benchmark::Default_Style: 87 Benchmark::timethese: 83 Benchmark::time: 78 Benchmark::Cache: 79 Benchmark::cpu_p: 79 Benchmark::Debug: 79 Benchmark::clearcache: 84 Benchmark::BEGIN: 79 Benchmark::Do_Cache: 82 Benchmark::runloop: 81 Benchmark::timedebug: 83 Benchmark::real: 78 Benchmark::usage: 79 Benchmark::clearallcache: 87 Benchmark::timeit: 80 Benchmark::EXPORT: 80 Benchmark::croak: 79 Benchmark::import: 80 Benchmark::disablecache: 86 Benchmark::enablecache: 85 Benchmark::Min_Count: 83 Benchmark::carp: 78 Benchmark::Default_Format: 88 Benchmark::VERSION: 81 Benchmark::timethis::: 84 Benchmark::mytime: 80 c:: 203

    I make no guess as to the absolute accuracy of the numbers, but they ought to reflect changes fairly well.

    As all the best lazy authors say; turning that into a module is left as an exercise :)

    Note: I eliminated a bunch "Devel::Size: Calculated sizes for compiled regexes are incompatible, and probably always will be" warnings, as I don't think they are produced by newer versions of DEvel::Size (though the situation they document hasn't change AFAIK.).


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
    In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit
Re: way to find module memory usage?
by Discipulus (Abbot) on Jun 06, 2018 at 21:12 UTC
    Hello perl-diddler,

    very intresting question.. I bet with no answer ;=) Have you already looked if Devel::Size can be handy in your case? See also Memory Profiling even if is an old thread.

    Also Memchmark is very old but it seems it is still passing tests on 5.28 so..

    Good luck and share your findings!

    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

      Devel::MAT is also worth a look.

      Edit: Although I don't think it will measure the code footprint, so is probably not what you're after.

      Yeah, Devel::Size and Devel::SizeMe were first I ran into ... but only work for data/vars.
Re: way to find module memory usage?
by shmem (Chancellor) on Jun 07, 2018 at 13:20 UTC
Re: way to find module memory usage? (Popey b size)
by Anonymous Monk on Jun 07, 2018 at 10:46 UTC
    Since 1999 B::TerseSize - Printing info about ops and their (estimated) size

      According to CPAN Testers matrix, B::TerseSize (or rather, the whole B-Size distribution) hasn't had a successful install on versions of perl later than 5.8.9, due to compilation errors in the XS. Unless you know how to get it to compile and install on a modern perl, are you sure it's really worth recommending?

Re: way to find module memory usage?
by Anonymous Monk on Jun 11, 2018 at 01:27 UTC
    Unfortunately for all of us, these are always virtual sizes ... never the actual metrics which drive the actual virtual-memory subsystems page by page ...

      Perhaps you might expand on what you mean or how it might help solve or at least approach the problem. Some citation? Some exploration code? Some way to do it with Excel or COBOL or RPC…?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1216051]
Front-paged by Discipulus
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (5)
As of 2022-01-18 13:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (53 votes). Check out past polls.

    Notices?