tobias_hofer has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks

I am facing an increasing memory usage over the time (during the application-run) and i am not shure what could be the root-cause.
Ok.. I got here a test-system written in perl (not by me) which is loading test-code (also written in perl)more or less dynamically like this:
 $para_return = &{ $tc_module."::".$para_init_func } ();

[download]
The test-system itself and the test-code are using the same modules for accessing global variables.
However, some of those modules have some globals which will be accessed/filled during the test run - in example:
our $VERDICT = VERDICT_NONE;
our @ACTUAL_ERROR_CODES = ();
our %TC_PARAMETER = ();
our %TC_PRJ_PARAMETER = ();
our @TC_HTML_TEXT = ();
our $REPEAT_TC_FLAG = 0;
my $ALL_ERR_CODES = {};
my $ErrorHandler = ITF::ErrorHandler->new();
my @SYS_FAIL_ERRORS = qw( 111 112 113 100 160 );
my $current_loglevel;
my $initial_loglevel;
my $current_TC_ID;
my %All_ZipFile_to_Handle;
my %All_ZipHandle_to_File;
my %Alive_Hosts;
my %PUBLIC_VARS;
our %EVAL_COLLECTION;

[download]
Now the badest. Some of these modules include each other, and modules are loaded several times.

Can modules be loaded several times?

Now my wild guess, in case modules can be loaded several times the global vars stay as they are but during the "use" the local vars and objects are created each time the module is loaded. Thus the memory increases - I guess ?

Am I right or completely wrong?

Is there in perl something like the include-protection in C ?

Replies are listed 'Best First'.
Re: Memory usage of array and hashe variables by multiple inclusion.
by kcott (Archbishop) on Jul 02, 2013 at 11:25 UTC

    G'day tobias_hofer,

    "Can modules be loaded several times?"

    require says:

    require demands that a library file be included if it hasn't already been included.

    and use says:

    It is exactly equivalent to 

    BEGIN { require Module; Module->import( LIST ); }

    [download]

    So, assuming you're using use or require to load the modules, the answer's no.

    You may find Test::LeakTrace helpful.

    -- Ken

[reply]
[d/l]
[select]
      Hello Ken,

      Thanks a lot!
      So I can exclude this as root-cause of the memory consumption.
      ;-)
[reply]
Re: Memory usage of array and hashe variables by multiple inclusion.
by flexvault (Monsignor) on Jul 02, 2013 at 13:16 UTC

    tobias_hofer,

      I am facing an increasing memory usage over the time (during the application-run) and i am not sure what could be the root-cause.

    When I have this problem, it usually is a global variable that continues to grow. In some cases, I use a subroutine that I call every so many seconds to show the memory status of the variables. Something like this (Note: I did this from memory so it may not compile asis). 

    sub ShowGrowth
{   use Devel::Size qw(total_size);
    our ( $BigScalar, %BigHash, @BigArray );
    my $nokeys = keys ( %BigHash ); my $noitems = $#BigArray;
    print $DLOG "####### Devel::Size #########\n";
    print $DLOG "\$BigScalar: ",total_size(\$BigScalar),"\n";
    print $DLOG "\%BigHash: ",total_size(\%BigHash),"  No of keys: $no
+keys\n";
    print $DLOG "\@BigArray: ",total_size(\@BigArray), " No of element
+s: $noitems\n";
}

    [download]
    You may want to add the actual memory location to the log. I have learned the hard way that the variable that I think I'm clearing may be the wrong one. To get the memory address: 
        print $DLOG "\$BigScalar: ",total_size(\$BigScalar)," Addr: ",\$Bi
+gScalar,"\n";

    [download]
    Leaking memory problems are never easy to find, but this code may give you some ideas on how to find the problem.

    Regards...Ed

    "Well done is better than well said." - Benjamin Franklin

[reply]
[d/l]
[select]
      Thanks a lot! I am now checking the size of the globals. They seem to increase quite fast due to logging

      Best Regards!
      Tobias
[reply]
Re: Memory usage of array and hashe variables by multiple inclusion.
by locked_user sundialsvc4 (Abbot) on Jul 02, 2013 at 13:51 UTC

    A common cause of memory-leaks in Perl are circular references, whereby references among objects form a “loop” such that the reference-count to the members of the structure never returns to zero and so the memory never gets reclaimed.

    Having “one thing in more than one place” is not a problem ... that’s just an object with multiple references to it ... so long as you arrange things such that, when an object is no longer required, all references to it go away.

    The “virtual memory size” as seen by an operating system does not know anything about the in-use/free status of individual objects as seen by Perl.

Back to Seekers of Perl Wisdom