Re^3: memory leaks with threads

The thing is, I'm not at all convinced that your posted code is evidence of a memory leak.

On the surface, it looks like it is because it looks like you are recording the maximum number of concurrent threads, but the way you are doing it is flawed.

If your main thread gets a timeslice between this line:

        $count = $threadscount;
[download]

and this line:

        if ( $count>$maxthreads ){
[download]

Then it could easily create 300 more threads, that would increment $threadscount, but they would not be reflected in $count, so you would not be recording the true maximum concurrent threads.

Basically, there is nothing in your code to prevent you from creating more than 300 threads, so the fact that the maximum thread count never gets higher than 300 is highly suspicious.

In essence, I don't believe you are demonstrating a memory leak. I think the amount of memory used is simply a multiplication factor resulting from the fact that there is no constraint on the concurrent number of threads, and at various times, this number is rising well above the 300 number that your code is reporting.

Would you please run the following code, which explicitly constrains the maximum number of threads, whilst creating and destroying threads at a high rate and monitor the memory usage using top and post your findings?

use threads;
use threads::shared;

our $S ||= 1;

my $t:shared = 0;

$|++;

while( 1 ){
    printf"\r$m\t%d\t", do{ lock $t; $t };
    if( do{ lock $t; $t } < 10 ) {
        $m++;
        threads->create(
            sub{
                { lock $t; $t++ }
                select undef, undef, undef, $S;
                { lock $t; $t-- }
            }
        )->detach;
     }
     else{
        select undef, undef, undef, 0.01;
    }
}
[download]

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Comment on Re^3: memory leaks with threads Select or Download Code

Replies are listed 'Best First'.
Re^4: memory leaks with threads by misc (Friar) on Jul 09, 2007 at 17:39 UTC
I ran your code, at the start it has a res memory consumption of around 12MB. After a minute this is growing to 18MB, I'd also say the process's memory consumption is growing in random steps. With perl5.9.5 it uses less memory and the growth is a bit slower, but still there. I didn't understand what you wrote about my code, since threadscount is locked until it's increased and $count=$threadscount. Could you perhaps explain this ? I found however an other flaw, it would be possible that the mainthread obtains it lock on $threadscount BEFORE the first thread increases it. Since the main thread waits in it's main loop until $threadscount is 0, there could (possibly) be created more than 300 threads. But (hopefully) fixing this didn't change anything.	[reply]
Re^5: memory leaks with threads by BrowserUk (Patriarch) on Jul 09, 2007 at 18:24 UTC
I didn't understand what you wrote about my code, since threadscount is locked until it's increased and $count=$threadscount. Could you perhaps explain this ? If main creates a new thread, then it exists, and therefore is consuming memory, before it can possibly increment `$threadscount`. So, if a thread gets a lock on `$threadcount`, and then gets swapped out holding that lock, and main gets a timeslice, then main will go into a tight loop creating 300 threads. Each of those threads. when it gets a timeslice, will attempt to get a lock on `$threadscount`, but there is still a lock held by another thread, so they will block. The result is, you've just created 300 new threads that each occupy memory, but that will never show up in the `$maxthreads` count, because they are all blocked from incrementing `$threadscount`. Actually, that is only one of several scenarios in your code that would lead to `$maxthreads` not reflecting the true situation. Could you try this also? It absolutely guarentees that there are never more that `$MAX` threads (default=10) threads running concurrently. It uses the thread id of the latest thread created to monitor how many threads have been created so far. On my system with `$MAX` set to 10, the memory usage wobbles a (very) little either side of 6MB, but even after 100,000+ creation/join cycles, it never displays any sign of long term memory growth. `use threads; use threads::shared; our $MAX \|\|= 10; $\|++; my @threads; while( 1 ) { push @threads, threads->create( sub{ 1; } ) while @threads < $MAX; printf "\rCreated so far; %d\t", $threads[ -1 ]->tid; $_->join for @threads; @threads = (); }` [download] What do you see? Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l] [select]
Re^6: memory leaks with threads by misc (Friar) on Jul 09, 2007 at 18:49 UTC
Thanks for your explanation. The last script however grows very fast and constantly, after 1 minute it has about 30MB resident memory consumption. I tried with perl 5.8.8 and perl 5.9.5. I also checked /proc/$PID/status The number of threads is not growing above $MAX. Is there a possiblity that this is somehow related to my dual core processor ?	[reply]
Re^7: memory leaks with threads by BrowserUk (Patriarch) on Jul 09, 2007 at 19:25 UTC
Re^8: memory leaks with threads by misc (Friar) on Jul 09, 2007 at 20:07 UTC
Some notes below your chosen depth have not been shown here
Re^8: memory leaks with threads (race of 1) by tye (Sage) on Jul 09, 2007 at 21:09 UTC
Some notes below your chosen depth have not been shown here
Re^4: memory leaks with threads by royer_k (Initiate) on Sep 24, 2007 at 10:55 UTC
I have tried running this on an intel arch and amd64 arch machine and found it does leak over time. We have a severe memory leak issue that appears to be related to detached threads in some of our code. Running the 5.8.8 Your test when run on amd64 platform took 36M after 10 minutes we're upto 63M and rising.... My laptop running Ubuntu feisty (i386) leaked at a slower rate... Yours, Karl Royer	[reply]
Re^5: memory leaks with threads by BrowserUk (Patriarch) on Sep 24, 2007 at 12:42 UTC
Quite possibly. As I mentioned here, the problem does not appear on my win32 system and as I do not use unix, I can do nothing about it. Someone with a unix system that demonstrates the problem will have to raise a perlbug and/or look for workarounds. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]