Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Enlightened Monks,

I am writing a Bioinformatics grid service in perl where I use various computers to process parts of a BLAST (bioinformatics algorithm file). My program creates a thread to send work to a node and as the node ends his work, so should the thread. Since the program will be dealing with many nodes, thus creating and destroying many threads, memory is a big deal. I have noticed that each time a create and destroy a thread, my precious megabytes seem to leak! Below may be found a simpler version where the leak occurs. Please help me find the light.

Thanks,
Paulo Carvalho

#!/usr/bin/perl-w

#Libs
use strict; use threads; use threads::shared;
my %thr; #Threads controling each node
my %thr_up;  share (%thr_up); #if thread finished job $thr_up shuould 
+be = 0


SMALL:for (1 .. 1000) {
    
    #Find a waiting thread
    unless ($thr_up{"FREE_HOST"}) {
        $thr_up{"FREE_HOST"} = 1; #thread working flag
        $thr{"FREE_HOST"} = threads->create("blast_thread");
        sleep(1);
        
        my @list = threads->list();
        $list[scalar(@list-1)]->join () if (scalar(@list) > 2);
        foreach my $l (@list) {print "$l\n";}
        
    }
    #Just for my surprise:
    print $thr{FREE_HOST}->tid, ",$thr{FREE_HOST} \n" if ($thr{FREE_HO
+ST}->tid);
}

sub blast_thread {

    $thr_up{FREE_HOST} = 0;
    print "Server UP\n";
    return;
}

############################

[download]

Replies are listed 'Best First'.
Re: A Memory Leak in Perl iThreads?
by liz (Monsignor) on Mar 21, 2004 at 14:50 UTC
    I don't think you're experiencing a leak per se.

    On my iBook starting 100 threads without loading any additional Perl modules apart from "use threads" costs more than 31 Mbyte of memory. You might want to check this for yourself with my Benchmark::Thread::Size module like this: 

    $ perl -MBenchmark::Thread::Size=refonly
  #   (ref)                                                           
+ 
  0    1728
  1    2101
  2    2398
  5    3288    
 10    4780    
 20    7753
 50   16685
100   31572

    [download]
    Your sample code seems to want to start 1000 threads. Be prepared to have at least 315 Mbyte of memory available for that. Unfortunately, in my experience, any serious program very quickly takes at least 1 Mbyte per thread. So you better have a lot of RAM available if you want to start 1000 threads that way.

    You might also be interested in an article that I wrote about this earlier.

    Liz

[reply]
[d/l]
      Even though the program seems to start many threads, it only keeps three threads open, joining the other threads, and even though the memory it uses keeps going up and up. Try to copy and paste the code in your PC and watch your memory go up even though you are only with 4 threads open.
      Thanks,
      Paulo C.
[reply]
        Change: 
        $thr{"FREE_HOST"} = threads->create("blast_thread");

        [download]
        to: 
        threads->create("blast_thread");
$thr{"FREE_HOST"} = 1;

        [download]
        Seems you've stumbled on yet another way to make threaded Perl leak. Judging by the amount of memory leaked, it would seem that your way of doing it, doesn't release the memory used by the thread.

        I'll be filing a bug report for this to p5p. Until then, I would say: don't do it that way. Why aren't you using a shared scalar for the flag?

        Also, if you want to check whether a thread is still running, you might want to check out Thread::Running. And possibly Thread::Pool might also be of interest to you.

        Liz

[reply]
[d/l]
[select]
Re: A Memory Leak in Perl iThreads?
by BrowserUk (Patriarch) on Mar 21, 2004 at 17:17 UTC

    Creating and destroying hundreds of threads, especially if they are short lived) is (almost) never the right way to do things. This is doubly true for perl's iThreads on win32 which are far heavier that native threads.

    Re-jigging your app so that you start a pool of threads and re-use them once they have completed their last task is often quite easy to do.

    Take a look at Thread::Queue for a simple mechanism for supplying work to your thread pool. Essentially all that is needed is a wrapper around your blast_thread sub that takes an item of work off of a shared queue and processes it, then loops back and gets another. When the queue is empty, the thread can die. If you need to return values from the threads back to your main process, then a second queue written by the threads and read in a loop by the main process once it has started the thread pool is simple mechanism to put in place.

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
[reply]
[d/l]
      Ok.. I just found out a work around here, but I would like to inform you of another new discovery I have just made. OPENDIR is not thread safe... I have a sub :
      #sub get_files {
#    
#    my ($work_dir, $ext) = @_;
#    opendir (FILES, "$work_dir");
#    my @to_go = grep {/\.$ext$/} readdir(FILES);
#    close(FILES);
#    return (@to_go);
#    
#}

      [download]
      and I cant use it any more in my threaded program in windows, or else I get a Windows error!!!, this sub is not used by threads, but any way, its fatal... there goes another great perl bug to submit!

      Best wishes,
      Paulo C.

      P.S.I already did a workaround in the sub using @files = <*>;
[reply]
[d/l]
Re: A Memory Leak in Perl iThreads?
by Old_Gray_Bear (Bishop) on Mar 21, 2004 at 13:18 UTC
    Could you check to see what verion of Perl you are using? If I recall correctly, there were some severe memory issues with Perl 5.8.0 that were addressed (at least partly) in the 5.8.1 release.

    ----
    I Go Back to Sleep, Now.

    OGB

[reply]
      I am using activestate perl 5.8.3
      Thanks,
      Paulo Carvalho
[reply]
Re: A Memory Leak in Perl iThreads?
by cormac (Acolyte) on Mar 21, 2004 at 18:57 UTC
    You do not want to do this. Use POE.
    -
    Real men use curlies.
[reply]

Back to Seekers of Perl Wisdom