in reply to Re^3: memory leaks with threads
in thread memory leaks with threads

I ran your code, at the start it has a res memory consumption of around 12MB.
After a minute this is growing to 18MB,
I'd also say the process's memory consumption is growing in random steps.
With perl5.9.5 it uses less memory and the growth is a bit slower, but still there.

I didn't understand what you wrote about my code, since threadscount is locked until it's increased and $count=$threadscount.
Could you perhaps explain this ?

I found however an other flaw, it would be possible that the mainthread obtains it lock on $threadscount BEFORE the first thread increases it.
Since the main thread waits in it's main loop until $threadscount is 0, there could (possibly) be created more than 300 threads.
But (hopefully) fixing this didn't change anything.

Replies are listed 'Best First'.
Re^5: memory leaks with threads
by BrowserUk (Patriarch) on Jul 09, 2007 at 18:24 UTC
    I didn't understand what you wrote about my code, since threadscount is locked until it's increased and $count=$threadscount. Could you perhaps explain this ?

    If main creates a new thread, then it exists, and therefore is consuming memory, before it can possibly increment $threadscount. So, if a thread gets a lock on $threadcount, and then gets swapped out holding that lock, and main gets a timeslice, then main will go into a tight loop creating 300 threads. Each of those threads. when it gets a timeslice, will attempt to get a lock on $threadscount, but there is still a lock held by another thread, so they will block.

    The result is, you've just created 300 new threads that each occupy memory, but that will never show up in the $maxthreads count, because they are all blocked from incrementing $threadscount.

    Actually, that is only one of several scenarios in your code that would lead to $maxthreads not reflecting the true situation.

    Could you try this also? It absolutely guarentees that there are never more that $MAX threads (default=10) threads running concurrently. It uses the thread id of the latest thread created to monitor how many threads have been created so far.

    On my system with $MAX set to 10, the memory usage wobbles a (very) little either side of 6MB, but even after 100,000+ creation/join cycles, it never displays any sign of long term memory growth. 

    use threads;
use threads::shared;

our $MAX ||= 10;

$|++;

my @threads;
while( 1 ) {
    push @threads, threads->create( sub{ 1; } ) while @threads < $MAX;
    printf "\rCreated so far; %d\t", $threads[ -1 ]->tid;
    $_->join for @threads;
    @threads = ();
}

    [download]

    What do you see?

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
      Thanks for your explanation.
      The last script however grows very fast and constantly, after 1 minute it has about 30MB resident memory consumption.
      I tried with perl 5.8.8 and perl 5.9.5.

      I also checked /proc/$PID/status
      The number of threads is not growing above $MAX.

      Is there a possiblity that this is somehow related to my dual core processor ?
[reply]

        Hm. This appears to be a platform specific problem then, because I left the last script running and after close to 1/4 million thread creation/destroy cycles, the memory use is still rock steady: 

        C:\test>td-threads
Created so far; 212710


Image Name                   PID Session Name     Session#    Mem Usag
+e
========================= ====== ================ ======== ===========
+=
tperl.exe                   2284                         0      6,456 
+K

c:\test>tperl -v

This is perl, v5.8.6 built for MSWin32-x86-multi-thread
(with 3 registered patches, see perl -V for more detail)
[...]

Binary build 811 provided by ActiveState Corp. http://www.ActiveState.
+com
ActiveState is a division of Sophos.
Built Dec 13 2004 09:52:01
[...]

        [download]

        Which OS are you running? Maybe you should raise a perlbug against 5.9.5.

        Is there a possibility that this is somehow related to my dual core processor ?

        My gut says no, but I don't have a dual core to with which to verify my gut one way or another.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom