Elijah has asked for the wisdom of the Perl Monks concerning the following question:

How would one go about returning all system resources used by a thread to the OS? In a standard threaded model there will generally be some sort of looping function spawning numerous threads then a call to join after the completion of the looping function to ensure all threads run their natural course.

However if a thread returns successfully while the looping functions is still looping the thread will block essentially holding on to it's resources (mainly memory is my concern).

Now if I implement say some sort of load splitting function in the mix to try an elliviate this and make a call to join() after each callback to my load spliting function all the previous joined and returned threads still hold their resources until the entire main thread exists.

I am not sure if I am properly conveying what I want to with words so I will try to do it with code: 

#!/usr/bin/perl -w

use strict;
use threads;

my $count = 1;
for (1..10) {
   my $thread = threads->create(\&threadsub, $count);
   $count++;
   sleep(2);
}

for (threads->list()) {
   eval{$_->join()};
   print $_->tid()." has returned!\n";
}

sub threadsub {
   my $cnt = shift;
   print "I am thread $cnt!\n";
   sleep(5);
   print "returning from thread $cnt...\n";
}

[download]

Now if you run the above code you will see that the app will tell you it is returning from each thread real time but will not report a successful join and completion until the entire for loop is complete then from this point on it will report a return real time.

If I try to work around this and create a subroutine to join each thread passed to it and then make a call in the for loop after each thread creation then two things will happen that I do not want to happen. First being that the app will block waiting for a return from each thread one at a time and second even after a thread is returned the resources (memory) used by that thread will still not be released back to the OS probably due to the fact that each thread runs in the same memory space as the main thread..

Is there any way I can get around this so I can have the thread returns it's memory used to the OS after a successful return/join?


www.perlskripts.com

Replies are listed 'Best First'.
Re: release threads resources?
by dave_the_m (Monsignor) on Oct 11, 2005 at 09:24 UTC
    I'm not really sure I understand what you are asking, but note that a thread's memory resources are returned to perl after it is joined, but the memory isn't returned to the OS until perl quits. If you don't want to join() each thread, you can just detach() them just after creation instead, although then its then harder for the main thread to determine when all its childen have finished.

    Dave.

[reply]

      The problem with join is that there is no equivalent to the WNOHANG parameter of waitpid, so you cannot join threads as they terminate. Rather, you have to join them in some order and if the first thread handle you attempt to join is the last to finish, all the others remain consuming process resources until it terminates.

      For start-run-die-start another style algorithms, as are common with forking servers, it makes it difficult to control resource consumption by limiting the number of concurrent threads to some fixed maximum by only starting the next thread when one terminates. That's why I tend to detach the threads and use a shared var to track how many are still running, but if you need to retrieve their return values, you're stuck.

      That said, Perl's threads are sufficiently heavyweight that using a long running pool of looping threads that take new work items from a queue is generally a better option anyway, but the absence of a non-blocking join is a pain for some things.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
[reply]
      I have thought about and tried the detach method also since the return of each thread is not important but the memory issues still exist with the detach method as they did with the join method.

      www.perlskripts.com
[reply]

        As stated above, thread memory is returned to the process when the thread terminates, not the OS. The memory becomes a part of the free memory pool(s) that new program elements, both code and data will be allocated from as needed.

        What that means, is that if any of your existing threads allocate scalars, or arrays or instantiate new instance of classes etc. etc., then the memory from terminated threads is recycled to provide it. If you start a new thread, then that memory will be used to provide for that new thread.

        At the very extreme, if your threaded perl process terminated threads and never needed to allocate another bean of memory, and other processes on your system continued to call for more and more memory until the only "free memory" in the system was that freed from the terminated threads within your process, then the OS would swap that free memory to disk, and the other process would be allocated what it needed.

        That called swapping--and everyone knows swapping is bad right?-- but the clever bit is that if your process never attempts to re-use that swapped out free (virtual) memory, then it will just stay on disk and so it's equivalent of real memory will be available to any and every other process in the system anyway. Swapping is only bad if it happens to memory that is in constant use and results in thrashing.

        So, what is the "memory issue" you have with Perl's threads?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
[reply]

Back to Seekers of Perl Wisdom