in reply to Re^2: Threads question
in thread Threads question

Well I ran your code on linux( upping the count to 5000), and it starts out at about 4.4% mem (I have a gig of ram). After 2 minutes, it was up to 12.4 % ram. That is roughly a doubling of mem every minute...... that's what I'm talking about.

Also your code is quite simple, meaning it uses modules that "probably" are quite thread safe. I would next start trying to run something like WWW::Mechanize, or LWP::UserAgent in the threads, and see what they add to the garbage leftovers. Maybe Win32 does a better job with garbage collection than linux?

I don't understand what you mean by this? Shared vars are tied. Every time you reference one, the current value is retrieved from the master copy.

In a simple example you are right. I'm talking about (and I should have mentioned it) using GUI's with threads. Gui's will allow you to tie, but that won't work across threads. Here is a simple example, the count is not automatically reflected to the thread. 

#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;
use Tk;

my $count:shared=0;
my $thread = threads->new( \&launch_thread )->detach;

my $mw = MainWindow->new();
$mw->geometry("600x400+100+100");

my $l1 = $mw->Label(-textvariable => \$count)->pack();

MainLoop;

sub launch_thread {

   while (1){
       $count++;
       print "$count\n";
       sleep 1;
     }
}

[download]

I'm not very good with a simple Tie, but the following dosn't trigger the fetch or store, in a non-gui script. 

#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;
use Tie::Watch;

my $count:shared=0;
my $thread = threads->new( \&launch_thread )->detach;

my $watch = Tie::Watch->new(
   -variable => \$count,
#   -debug    => 1,
   -fetch    => \&fetch,
   -store    => \&store,
   -destroy  => sub {print "Final value=$count.\n"},
);


while(1){}


sub launch_thread {

   while (1){
       $count++;
       print "$count\n";
       sleep 1;
     }
}


sub store {
   print "changed\n";
}

sub fetch{
   print "fetched @_\n";

}

[download]

Maybe you can make the simple Tie report a store?

I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Replies are listed 'Best First'.
Re^4: Threads question
by BrowserUk (Patriarch) on Jun 29, 2007 at 17:40 UTC
    Well I ran your code on linux( upping the count to 5000), and it starts out at about 4.4% mem (I have a gig of ram). After 2 minutes, it was up to 12.4 % ram.

    Not unexpected. As there is no control on the number of threads running concurrently, you have to wait for the system to reach a 'steady state' as far as the number of threads running is concerned--ie. wait to the point that new threads are being started at teh same rate the old ones are dieing--before you can no longer attribute the growth to just more concurrent threads. Otherwise you have to wait until it ends and see what the final usage is.

    I should write a better test. One that maintains a steady number of concurrent threads by starting a new only when an old one dies after a set number. That would allow a better picture.

    Also your code is quite simple, meaning it uses modules that "probably" are quite thread safe.

    Um. Loading non-threadsafe modules in threads and not expecting problems is a bit naive :)--but I get your point. When I write a better test, I'll try and load some more complex, heavy modules.

    How about you try having Tk watch a tied variable--without threads. I guessing that it won't be able to successfully tie a tie (Windsor knot anyone? :). That is, I think the problem may not be threads::shared per se, but simply that tieing a tied variable doesn't work. Essentially the same reason that you can't share a tied variable.

    I'm not sure how you would verify that. It kinda hard to modify the first level of tie whilst the second level watches without using threads? Maybe you could set up a timer (after or repeat) to modify a simple tied scalar, and then pass that tied scalar to as the watched variable to 

    tie my $count, 'Tie::Scalar';
...
$mw->repeat( sub { $count++ }, 1000 );
...
my $l1 = $mw->Label(-textvariable => \$count)->pack();

    [download]

    I'd try it myself, but my Tk install is screwed ever since I tried install tcl/tk. I never got around to trying to sort that mess out.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      Well you seem to claim the "memory gain" is acceptable. I have found sometimes that is true, and sometimes not. Of course a 200k gain in a one-shot script is acceptable in most cases. A doubling of mem every hour in a long-running script, is not.

      The intent of my original post was to point out that the problem exists, and it can get complex (as thread code complexity rises). So I still say "watch your memory especially when objects are used in the thread code".

      As far as the tie problem goes, it really is a mute point, because when you monitor the thread's shared variables you need a loop or timer to read them. But it is a tripping point for new threads programmers.

      I did whip out an old Tk program to test it, Tk has a Trace module, to setup such ties. If you notice in the program below, the needle will not move with the Trace, UNLESS the main program explicitly reads(or changes) the variable. Uncomment the repeat statement to make it work. 

      #!/usr/bin/perl
use warnings;
use strict;
use threads;
use threads::shared;

my $v:shared=0;
my $thread = threads->new( \&launch_thread )->detach;

package Tk;
use Tk::Trace;

package main;
use Tk;
use constant PI => 3.1415926;

my $mw = MainWindow->new;

$mw->fontCreate('medium',
   -family=>'courier',
   -weight=>'bold',
   -size=>int(-14*14/10));

my $c = $mw->Canvas(
              -width => 200,
          -height => 110,
          -bd => 2,
          -relief => 'sunken',
          -background => 'black',
          )->pack;
          
$c->createLine(100,100,10,100,
          -tags => ['needle'],
      -arrow => 'last',
      -width => 5,
      -fill => 'hotpink',
       );

my $gauge = $c->createArc(
        10,10, 190,190,
        -start => 0,
    -style => 'arc',
        -width => 5,
        -extent => 180,
    -outline => 'skyblue',
        -tags => ['gauge'],
);

my $hub = $c->createArc(
        90,95, 110,115,
        -start => 0,
        -extent => 180,
    -fill => 'lightgreen',
        -tags => ['hub'],
);


$mw->traceVariable(\$v, 'w' => [\&update_meter]);

#uncomment the following line to make the tie work
#interestingly {$v = $v} will not do it
#$mw->repeat(50,sub{ $v += 0 });


my $text = $c->createText( 
      100,50,
       -text  => $v,
       -font  => 'medium',
       -fill  => 'yellow',
       -anchor => 's',
       -tags => ['text']
 );


$c->raise('needle','text');
$c->raise('hub','needle');

MainLoop;

sub update_meter {
   my($index, $value) = @_;
  if($value <= 0){$value = 0 }
  if($value >= 100){$value = 100}
    
    my $pos = $value / 100;
    my $x = 100.0 - 90.0 * (cos( $pos * PI ));
    my $y = 100.0 - 90.0 * (sin( $pos * PI ));
    $c->coords('needle', 100,100, $x, $y);
    $c->itemconfigure($text,-text => $value);
return $value;
}


sub launch_thread{

while(1){
  $v = 50 + int rand 50;
  print "$v\n";
  select(undef,undef,undef,.5);
 }

}

      [download]
      In Tk, there is a way with Tk::Trace to force a tie, but I havn't tried it across threads.

      So my general point is that threads in Perl is NOT as clean as threads in c, and "sometimes" a thread-reuse scheme is needed. The tie problem is really not worth trying to work out, since a timer is so easy to setup

      I'm not really a human, but I play one on earth. Cogito ergo sum a bum
[reply]
[d/l]
        Well you seem to claim the "memory gain" is acceptable.

        Sorry, I was unclear. The point I was trying to make is that a program with 11 threads running will (inevitably) use more memory than the same program with 10 threads running.

        So in order to determine whether there is any net growth as a result of continually replacing old threads with new ones, you need to measure the memory at points where there are the same number of threads running. Or wait for the cycle to complete and see if there was any net growth from beginning to end.

        On my system that was easy to check, as memory allocated to the threads is returned to the OS when the thread dies. On Linux, where memory is returned to the process free memory pool, but not to the OS, it's impossible to tell whether the memory allocated to the process is available for use by the next thread started or not.

        On releases of Perl prior to 5.8.6, it was easy to demonstrate that the cleanup of memory, when a thread dies, was not total. This was easily demonstrated by continuously creating short lived threads as fast as possible, through a steady and inexorable growth in the memory allocated to the process. With 5.8.8, that behaviour ceased (on win32 at least).

        It may still be there to a lesser degree (ie. a smaller leak), but it will require a better designed mousetrap to demonstrate it. I'll have a go at adapting my program.

        On the watch variable: My point was that I believe that it would be impossible to watch any type of tied variable using Tk. Which obviously includes shared vars as they are tied. So the problem is not specific to shared vars, but it shows up when you use threads, because it's so much easier to concurrently alter the value of a (watched) tied variable, when you have shared variables & threads running.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom