in reply to memory leaks with threads

Which versions of Perl and threads are you using?


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^2: memory leaks with threads
by misc (Friar) on Jul 09, 2007 at 16:15 UTC
    perl 5.8.8,
    threads 1.07
    threads::shared 0.94

    I just upgraded threads and threads::shared to 1.63 and 1.12,
    but this didn't change anything.

    At the moment I'm trying to compile perl 5.9.5..

      The thing is, I'm not at all convinced that your posted code is evidence of a memory leak.

      On the surface, it looks like it is because it looks like you are recording the maximum number of concurrent threads, but the way you are doing it is flawed.

      If your main thread gets a timeslice between this line:

      $count = $threadscount;

      and this line:

      if ( $count>$maxthreads ){

      Then it could easily create 300 more threads, that would increment $threadscount, but they would not be reflected in $count, so you would not be recording the true maximum concurrent threads.

      Basically, there is nothing in your code to prevent you from creating more than 300 threads, so the fact that the maximum thread count never gets higher than 300 is highly suspicious.

      In essence, I don't believe you are demonstrating a memory leak. I think the amount of memory used is simply a multiplication factor resulting from the fact that there is no constraint on the concurrent number of threads, and at various times, this number is rising well above the 300 number that your code is reporting.

      Would you please run the following code, which explicitly constrains the maximum number of threads, whilst creating and destroying threads at a high rate and monitor the memory usage using top and post your findings?

      use threads; use threads::shared; our $S ||= 1; my $t:shared = 0; $|++; while( 1 ){ printf"\r$m\t%d\t", do{ lock $t; $t }; if( do{ lock $t; $t } < 10 ) { $m++; threads->create( sub{ { lock $t; $t++ } select undef, undef, undef, $S; { lock $t; $t-- } } )->detach; } else{ select undef, undef, undef, 0.01; } }

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        I ran your code, at the start it has a res memory consumption of around 12MB.
        After a minute this is growing to 18MB,
        I'd also say the process's memory consumption is growing in random steps.
        With perl5.9.5 it uses less memory and the growth is a bit slower, but still there.

        I didn't understand what you wrote about my code, since threadscount is locked until it's increased and $count=$threadscount.
        Could you perhaps explain this ?

        I found however an other flaw, it would be possible that the mainthread obtains it lock on $threadscount BEFORE the first thread increases it.
        Since the main thread waits in it's main loop until $threadscount is 0, there could (possibly) be created more than 300 threads.
        But (hopefully) fixing this didn't change anything.
        I have tried running this on an intel arch and amd64 arch machine and found it does leak over time. We have a severe memory leak issue that appears to be related to detached threads in some of our code. Running the 5.8.8 Your test when run on amd64 platform took 36M after 10 minutes we're upto 63M and rising.... My laptop running Ubuntu feisty (i386) leaked at a slower rate... Yours, Karl Royer