Re^3: Massive Perl Memory Leak

   $foo = 5;
   sub do_something { print $foo, $/; }
   sub main
   {
      do_something();
      {
        local $foo = 6;
        do_something();
      }
      do_something();
   }
   main();
[download]

The above code is a typical use of local. Internally, it is conceptually equivalent to the following code (shown for the innermost block only). You'd get 5, 6, 5 printed.

      {
        my $unnamed_backup_of_old_value_of_foo = $foo;
        $foo = 6;
        do_something();
        $foo = $unnamed_backup_of_old_value_of_foo;
      }
[download]

The actual storage of $foo is the first statement. It is what gets changed whenever you see assignments being done.

This of course can get very big and hairy with a huge hash of data to "backup" and "restore" in the equivalent %unnamed_backup_of_old_value_of_datahash. Also, deleting items from the local hash would not have much bearing, since the whole hash will just get tossed out and restored from the backup.

If you're doing local from multiple threads, well, you can see where your memory is going. Secondly, I can quite easily imagine a race condition where your main shared global hash is getting "restored" in the wrong order, leading to pandemonium.

--
[ e d @ h a l l e y . c c ]

Comment on Re^3: Massive Perl Memory Leak Select or Download Code

Replies are listed 'Best First'.
Re^4: Massive Perl Memory Leak by wagnerc (Sexton) on Jun 11, 2007 at 18:32 UTC
%datahash is initialized as nul since it's not used until we're already inside the forever loop. So when local reinstantiates it at each pass it "restores" it's value to that of the nul list, (). In my usage, I could replace it with %datahash = (); and the functionality would be identical. %datahash is also an unshared variable. No other threads can access it. So so long as Perl's innards are clean, no other thread can tamper with %datahash. I'm using the term thread global to mean it's global only within that one thread. To get program globals u have to share() them. Does anybody know of a way to "see" data/memory that doesn't have a varname pointing to it? To find out what "anonymous storage" is in use.	[reply]
Re^5: Massive Perl Memory Leak by BrowserUk (Patriarch) on Jun 11, 2007 at 19:50 UTC
I think you are barking up the wrong tree looking at %datahash as the cause of the problem. The following simple script emulated what you are describing and once the 101 threads are up and running, the memory usage is rock steady under both 5.8.6/AS811 with the AS delivered core version of threads and 5.8.8/AS817 with the latest cpan versions of threads & threads::shared. And that's relying upon Perl's garbage collection alone. More information needed. (I still suspect Data::Dump) #! perl -slw use threads; use threads::shared; print $threads::VERSION; print $threads::shared::VERSION; print $]; @th = map{ threads->new( sub{ while( 1 ){ local %h = ( 1 .. 10000 ); sleep 1 } } ) } 1 .. 100; $_->join for @th __END__ c:\test>junk9 Name "main::h" used only once: possible typo at c:\test\junk9.pl line +13. 1.05 0.92 5.008006 Terminating on signal SIGINT(2) c:\test>\AS817\perl\bin\perl5.8.8.exe junk9.pl Name "main::h" used only once: possible typo at junk9.pl line 13. 1.1201 0.99 5.008008 Terminating on signal SIGINT(2) [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^6: Massive Perl Memory Leak by wagnerc (Sexton) on Jun 13, 2007 at 20:08 UTC
Well I just restored %datahash usage to the rebuild of the script and it didn't leak. :) Kind of glad that's not it. I also just restored Data::Dump usage to the rebuild and am testing it now. We'll see if it goes down. I don't remember if I was using Data::Dump in the first place when I noticed the leak. Actually, yes. I remember now, I need Data::Dump to serialize the hash to pass it through a message queue to the database thread.	[reply]