Hm. This appears to be a platform specific problem then, because I left the last script running and after close to 1/4 million thread creation/destroy cycles, the memory use is still rock steady:

C:\test>td-threads
Created so far; 212710


Image Name                   PID Session Name     Session#    Mem Usag
+e
========================= ====== ================ ======== ===========
+=
tperl.exe                   2284                         0      6,456 
+K

c:\test>tperl -v

This is perl, v5.8.6 built for MSWin32-x86-multi-thread
(with 3 registered patches, see perl -V for more detail)
[...]

Binary build 811 provided by ActiveState Corp. http://www.ActiveState.
+com
ActiveState is a division of Sophos.
Built Dec 13 2004 09:52:01
[...]
[download]

Which OS are you running? Maybe you should raise a perlbug against 5.9.5.

Is there a possibility that this is somehow related to my dual core processor ?

My gut says no, but I don't have a dual core to with which to verify my gut one way or another.

I'm running Linux, 2.6.18, 32bit on a turion x2 tl-60.

I just ran a similar program written in c.
It doesn't grow (stays exactly at a res memory of 944B), also after 15 million threads created and destroyed.
So I can at least say it's not related to my pthreads library.

I'll however wait a bit before submitting a bug report, maybe someone else knows a solution.


#include <iostream>
#include <cstdlib>

using namespace std;

#include <pthread.h>
#include <unistd.h>
#include <sys/select.h>
#include <sys/time.h>

using namespace std;

static int threadscount = 0;
pthread_mutex_t threadscount_mutex = PTHREAD_MUTEX_INITIALIZER;



void* thread( void* data ){
        pthread_mutex_lock( &threadscount_mutex );
        threadscount--;
        //cout << threadscount << endl;
        pthread_mutex_unlock( &threadscount_mutex );
        pthread_exit(0);
}

int main(int argc, char *argv[]){
        int count = 0;
        int a;
        
        struct timeval timeout;

        while (1){
                for ( int a = 1; a <= 10; a++ ){
                        pthread_mutex_lock( &threadscount_mutex );
                        threadscount++;
                        pthread_mutex_unlock( &threadscount_mutex );
                        pthread_t  p_thread;       /* thread's structu
+re                     */
                        pthread_create(&p_thread, 0, thread, (void*)&a
+);
                        pthread_detach( p_thread );

                        count ++;
                }
                cout << "count: " << count << endl;
                int c;
                do {
                        timeout.tv_usec = 100;
                        timeout.tv_sec = 0;
                        select( 0, 0, 0, 0, &timeout );

                        pthread_mutex_lock( &threadscount_mutex );
                        c = threadscount;
                        pthread_mutex_unlock( &threadscount_mutex );
                    } while ( c > 0);
                        
        }
}
[download]

It's no great surprise that it's not the system apis that are at fault, but the perl source code on top of them.

I'll however wait a bit before submitting a bug report, maybe someone else knows a solution.

Okay, but if the last version of code I posted leaks that badly on 5.9.5, it's most unlikely that there is anything that can be done at Perl level (ie. in your code) to 'fix' the problem. You're going to have to wait at least until dave_the_m and co. at p5p track down and fix whatever is wrong. That's why I suggested the perlbug to ensure that a) it comes to their attention; b) they get all the relevant information they need.

Looking at the problem from a different angle. As you saw by comparing the speed and memory usage of pthreads at the C level, and ithreads at the perl level, ithreads are very heavy by comparison. That's easily explained by the benefits they give you--automatic memory management, explicitly shared, rather than implicitly shared variable spaces, and everything else that is Perl over C. And the work that is involved in achieving those--basically, starting a new interpreter and cloning the current environment each time you start a thread.

What all that is building up to, is that spawning threads for small amounts of work and then throwing them away--whilst very effective in C--is not the best way of tackling a threading problem using ithreads and perl. A better way is to use a pool of threads and re-use them.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Is there a possibility that this is somehow related to my dual core processor ?

My gut says no, but I don't have a dual core to with which to verify my gut one way or another.

I've seen that if one has a single core, then one can avoid many race conditions because only one thread ever runs at a time. I've seen several cases of code that works great "forever" on a single-core system but gets confused when run on a multi-processor or multi-core system. Losing track of resources and thus leaking memory is certainly a possible outcome of such race conditions.

It isn't terribly hard to test such either, just bind the process to a single processor / core and see if the memory leak goes away.

The leak being that dramatic vs. not at all is a rather stark difference to explain with a race condition, however, so a configuration difference seems more likely.

- tye        

I've seen that if one has a single core, then one can avoid many race conditions because only one thread ever runs at a time.

I completely agree that moving from single-cpu multi-threading to multi-cpu multi-threading often causes latent bugs to become glaringly obvious. Though I think this is more apparent with compiled-to-native code than ithreads, simply because you do not get the 'atomicity' of native machine instructions with Perl op-codes.

The basis of my gut feel is the lack of any great program level memory allocations, and the inherent simplicity of the OPs demo code.

Given the latest info from the OP, that creating & destroying threads consisting soley of sub{ 1; } causes fast, massive leaks on his system, I have to suspect that the leak is inherent in the implementation of the creation and freeing of interpreter structures on his platform.

The big difference between *nix and Win32 in this regard is that Win32 returns memory segments allocated by the creation of threads to the OS. So, even if during the creation of the internal structures required to create a new interpreter, there is some circular reference or other reference counting error, that on the *nix system means that some part of it does not get free'd back to the memory pool for re-use, on the win32 system, the entire kit'n'kaboodle gets given back to the OS, so reference counting errors do not persist.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."