using many threads and conserving stack size

danmcb has asked for the wisdom of the Perl Monks concerning the following question:

I have a test script which I am using to exercise an application. The test app creates a new thread each second, each thread waits for a random period before exiting. I want to be able to create as many threads as possible before hitting memory limits. This ought to be pretty much unlimited as threads end more or less as quickly as they are created, once things hit steady state

In practice, it seems that threads don't actually get cleaned up when the thread exits, and thread->create fails after a few hundred. I can improve this by twaeking the stack size, but I'd like to make it that the thread memory is really free after exiting.

Here is a trimmed down version of the code ...

#!/usr/bin/perl -w
use strict;
use threads;

for (my $i = 0; $i < 5000; $i++){
    my $thr = threads->create( \&_run_thread, $i );
    sleep 1;
}

# make sure all threads complete
while ( threads->list(threads::running) ){
        sleep 1;
    };
print "ok.\n";

sub _run_thread {
    my ($id) = @_;
    print "starting thread $id ...\n";
    sleep 4;
    print "exiting thread $id.\n";
    return;
}
[download]

In the real code, the sleep 4 is actually a random time, and of course the thread does other uninteresting stuff. When the code ends, the app prints a warning that threads were all finished but not joined and not detached. If I somehoe try to join the finished ones in the main loop, will I get back the memory from those?

(In practice setting stack size to 32k seems to make this not a real issue, but I'm ever curious ...)

Comment on using many threads and conserving stack size Download Code

Replies are listed 'Best First'.
Re: using many threads and conserving stack size by BrowserUk (Patriarch) on Nov 12, 2008 at 13:23 UTC
Add `$thr->detach;` after the create. On my system I reach a steady state memory usage of around 2.7 MB after a few seconds. That's with threads v1.71. (YMMV with earlier versions). Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^2: using many threads and conserving stack size by danmcb (Monk) on Nov 12, 2008 at 14:33 UTC
ah, there is a gotcha. If I detach every thread, then the script exits before all threads have actually exited. What I want is to get back the memory of the threads when they exit, but I want every thread to complete its sleep and then exit. That is really important for this application.	[reply]
Re^3: using many threads and conserving stack size by BrowserUk (Patriarch) on Nov 12, 2008 at 17:44 UTC
Sorry. I shouldn't post just before leaving. You're right. There is still no architected way to find out how many detached threads are still running. So, you need to implement some counting mechanism yourself. A simple shared scalar that you increment when the threads start and decrment when the threads end is simple and efficient. `#!/usr/bin/perl -w use strict; use threads; use threads::shared; my $running :shared = 0; for (my $i = 0; $i < 50; $i++){ my $thr = threads->create( \&_run_thread, $i ); $thr->detach; sleep 1; } sleep 1 while $running; print "ok.\n"; sub _run_thread { { lock $running; ++$running; } my ($id) = @_; print "starting thread $id ...\n"; sleep 4; print "exiting thread $id.\n"; { lock $running; --$running; } return; }` [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^3: using many threads and conserving stack size by gone2015 (Deacon) on Nov 12, 2008 at 16:00 UTC
What that comes down to getting the parent to know when a thread has finished. You could use `Thread::Queue`, and each thread can enqueue its id when it's finished. The parent could wait on that queue or poll it. Or you could poll `threads->list(threads::joinable)` for threads you can polish off by joining. You might do that before creating a new thread, I suppose.	[reply] [d/l] [select]
Re^4: using many threads and conserving stack size by danmcb (Monk) on Nov 12, 2008 at 16:04 UTC
Re^5: using many threads and conserving stack size by gone2015 (Deacon) on Nov 12, 2008 at 16:26 UTC
Some notes below your chosen depth have not been shown here
Re^2: using many threads and conserving stack size by danmcb (Monk) on Nov 12, 2008 at 13:58 UTC
thanks, that's all that was needed^H^H^H ... no it wasn't because it breaks the next part of the code that waits for all threads to finish before exiting - see below.	[reply]


There's more than one way to do things
	PerlMonks