in reply to Re: Massive Perl Memory Leak
in thread Massive Perl Memory Leak

Thanks for getting back to me. I'm using the local %datahash construct so I can avoid gotchas with passing references (but it seems I've run into another gothcha somewhere else). %datahash is essentially just a thread global and the local is there to automatically garbage collect it as the forever loop restarts. I can't use my because the variable wouldn't be visible to the descendant subroutines. Now correct me if I'm wrong, but isn't %datahash garbage collected the moment the forever loop loops? It's not creating *new* %datahashes and keeping the *old* %datahash's stashed away right? I'm manually walk deleting everything out of %datahash at the end of the loop so either way it shouldn't be building up memory.

Replies are listed 'Best First'.
Re^3: Massive Perl Memory Leak
by halley (Prior) on Jun 11, 2007 at 18:07 UTC
    $foo = 5; sub do_something { print $foo, $/; } sub main { do_something(); { local $foo = 6; do_something(); } do_something(); } main();
    The above code is a typical use of local. Internally, it is conceptually equivalent to the following code (shown for the innermost block only). You'd get 5, 6, 5 printed.
    { my $unnamed_backup_of_old_value_of_foo = $foo; $foo = 6; do_something(); $foo = $unnamed_backup_of_old_value_of_foo; }

    The actual storage of $foo is the first statement. It is what gets changed whenever you see assignments being done.

    This of course can get very big and hairy with a huge hash of data to "backup" and "restore" in the equivalent %unnamed_backup_of_old_value_of_datahash. Also, deleting items from the local hash would not have much bearing, since the whole hash will just get tossed out and restored from the backup.

    If you're doing local from multiple threads, well, you can see where your memory is going. Secondly, I can quite easily imagine a race condition where your main shared global hash is getting "restored" in the wrong order, leading to pandemonium.

    --
    [ e d @ h a l l e y . c c ]

      %datahash is initialized as nul since it's not used until we're already inside the forever loop. So when local reinstantiates it at each pass it "restores" it's value to that of the nul list, (). In my usage, I could replace it with %datahash = (); and the functionality would be identical. %datahash is also an unshared variable. No other threads can access it. So so long as Perl's innards are clean, no other thread can tamper with %datahash. I'm using the term thread global to mean it's global only within that one thread. To get program globals u have to share() them.

      Does anybody know of a way to "see" data/memory that doesn't have a varname pointing to it? To find out what "anonymous storage" is in use.

        I think you are barking up the wrong tree looking at %datahash as the cause of the problem.

        The following simple script emulated what you are describing and once the 101 threads are up and running, the memory usage is rock steady under both 5.8.6/AS811 with the AS delivered core version of threads and 5.8.8/AS817 with the latest cpan versions of threads & threads::shared. And that's relying upon Perl's garbage collection alone.

        More information needed. (I still suspect Data::Dump)

        #! perl -slw use threads; use threads::shared; print $threads::VERSION; print $threads::shared::VERSION; print $]; @th = map{ threads->new( sub{ while( 1 ){ local %h = ( 1 .. 10000 ); sleep 1 } } ) } 1 .. 100; $_->join for @th __END__ c:\test>junk9 Name "main::h" used only once: possible typo at c:\test\junk9.pl line +13. 1.05 0.92 5.008006 Terminating on signal SIGINT(2) c:\test>\AS817\perl\bin\perl5.8.8.exe junk9.pl Name "main::h" used only once: possible typo at junk9.pl line 13. 1.1201 0.99 5.008008 Terminating on signal SIGINT(2)

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re^3: Massive Perl Memory Leak
by GrandFather (Saint) on Jun 11, 2007 at 19:01 UTC

    I can't answer the garbage collection issue except to say what you probably already know: Perl deletes objects' memory when it is no longer referenced.

    However using global variables to "pass" information between subs is bad, bad, bad! Instead pass a reference to the descendant subs so it is clear where information is being used and possibly modified. If you find you are passing a lot of parameters consider using light weight object oriented techniques by simply blessing a hash into the current package then calling the descendant subs on the object:

    my $obj = bless {datahash => \%datahash, ...}; $obj->sub1 (); $obj->sub2 (); ... sub sub1 { my $self = shift; $self->sub3 (wibble => 3); } ... sub sub3 { my ($self, %params) = @_; for my $key (keys %{$self->{datahash}}) { my $value = $self->{datahash}{$key} * $params->{wibble}; ... } }

    and if you have trouble with getting references going ask! Writing nasty code to avoid learning a new and valuable technique will not reward you over the long run, and by the sound of it not even in the short run. You may find References quick reference helps.


    DWIM is Perl's answer to Gödel