in reply to Re^5: Threads question
in thread Threads question

Well you seem to claim the "memory gain" is acceptable.

Sorry, I was unclear. The point I was trying to make is that a program with 11 threads running will (inevitably) use more memory than the same program with 10 threads running.

So in order to determine whether there is any net growth as a result of continually replacing old threads with new ones, you need to measure the memory at points where there are the same number of threads running. Or wait for the cycle to complete and see if there was any net growth from beginning to end.

On my system that was easy to check, as memory allocated to the threads is returned to the OS when the thread dies. On Linux, where memory is returned to the process free memory pool, but not to the OS, it's impossible to tell whether the memory allocated to the process is available for use by the next thread started or not.

On releases of Perl prior to 5.8.6, it was easy to demonstrate that the cleanup of memory, when a thread dies, was not total. This was easily demonstrated by continuously creating short lived threads as fast as possible, through a steady and inexorable growth in the memory allocated to the process. With 5.8.8, that behaviour ceased (on win32 at least).

It may still be there to a lesser degree (ie. a smaller leak), but it will require a better designed mousetrap to demonstrate it. I'll have a go at adapting my program.

On the watch variable: My point was that I believe that it would be impossible to watch any type of tied variable using Tk. Which obviously includes shared vars as they are tied. So the problem is not specific to shared vars, but it shows up when you use threads, because it's so much easier to concurrently alter the value of a (watched) tied variable, when you have shared variables & threads running.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^7: Threads question
by zentara (Cardinal) on Jun 30, 2007 at 16:30 UTC
    I have to agree with you that it all works pretty well, that is why I like using threads. I did some more realistic testing and found that the threads were collecting garbage alot better than I thought they would. In this more realistic script, the memory holds steady. I don't know why my previous experience with threads showed otherwise..... maybe it was because I was using guis, (non-thread-safe Tk) or older Perl versions? Possibly I was leaving non-detached threads in an unjoined condition, OR had threads which were hanging somewhere in their code? 
    #!/usr/bin/perl
use warnings;
use strict;
use threads;
use threads::shared;

my $count = 0;

while (1) {
        $count++;
        print "thread $count started\n";         
        my $t = threads->new(\&do_my_thing, $count)->detach;
        sleep 5;
    }


sub do_my_thing {
    
    use LWP::Simple;
    use Data::Dumper;
    use HTML::TokeParser::Simple;
   
   my $get = 'http://google.com';
   my $page=get( $get );

   my $p = HTML::TokeParser::Simple->new( \$page );
    while ( my $token = $p->get_token ) {
       # This prints all text in an HTML doc (strips HTML)
           next unless $token->is_text;
       print $token->as_is;
      }

    my $val = shift;
    select(undef,undef,undef,.1);
    print "thread $val ended\n";         
}

    [download]
    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
[reply]
[d/l]

      You might also like to try this. On my system it settles down to a steady number of threads within 5 or 10 seconds and thereafter creates and destroys threads at the rate of about 7/second with the memory usage fluctuating a few Kb either side of 30MB.

      I left this running whilst I ate (be warned, it thrashes the cpu). The thread cycles had reached over 12000 by the time I returned and the memory usage was still locked at ~30MB. And that was with 5.8.6. If there is a leak there, it's an extremly slow one.

      If your processor is faster or slower than mine, you might need to tweak the sleep parameter within the thread to get it to achieve equilibrium quickly. 

      #! perl -slw
use strict;
use threads;
use Thread::Queue;
use threads::shared;
use Time::HiRes qw[ time ];

$|++;

our $N ||= 1000;
our $M ||= 10;

our $aClonedGlobal          = 12345;
our $aSharedGlobal :shared  = 12345;
my $aClonedLexical          = 12345;
my $aSharedLexical :shared  = 12345;

my $count :shared = 0;
my $running :shared = 0;
my $Q = new Thread::Queue;

sub thread {
    { lock $running; ++$running }
    my $tid = threads->self->tid;
    my( $some, $thread, $local, $vars ) = (12345) x 4;
    require Carp;
    require IO::Socket;
    require LWP::UserAgent;
    $Q->enqueue( threads->tid );
    sleep $M/10;
    { lock $running; --$running };
    return 1;
}

threads->create( \&thread ) for 1 .. $M;

my $start = time;
while( 1 ) {
    threads->object( $Q->dequeue )->join;
    threads->create( \&thread );
    printf "\r%d\t (%7.3f/sec)\t", ++$count, $count / ( time() - $star
+t );
    sleep 0;
}

      [download]
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom