in reply to Re^17: does threads (still) memleak?
in thread does threads (still) memleak?

I think I ruled out Dumper and STDERR cache: 
#!/usr/bin/perl

use warnings;
use strict;
use threads;
use threads::shared;

my %hash :shared;

sub dumper {
  my $h = shift;
  my $ret = "{\n";
  while (my ($k,$v) = each %$h) {
    $ret.= "\t$k => $v\n";
  }
  $ret.= "}\n";
  return $ret;
}

sub sub1 {
  my $str = '[s'.threads->tid()."]\n";
  $str.= dumper \%hash;
  $str.= '[e'.threads->tid()."]\n";
  return $str;
}

$hash{a} = 1;
my $th1 = threads->new('sub1');
$hash{b} = 2;
my $th2 = threads->new('sub1');

my $a = $th1->join();
my $b = $th2->join();

print STDERR $a;
print STDERR $b;

[download]
Which sometimes yields: 
[s1]
{
        a => 1
        b => 2
        a => 1
        b => 2
}
[e1]
[s2]
{
        a => 1
}
[e2]

[download]
And I just reproduced it on perl 5.8.5 32 and 64bit linux. Sometimes I also get: 
[s1]
{
        a => 1
        b => 2
        b => 2
}
[e1]
[s2]
{
        a => 1
        a => 1
        b => 2
}
[e2]

[download]
Then again, even though we see duplicate keys, their value is correct... Maybe the problem arises when trying to cycle through all keys of the hash at the exact same time its contents are being updated: If I add a select undef,undef,undef,0.01; between the two thread spawns, I never see duplicate keys. At the worst I see normal differences between $a and $b.

Replies are listed 'Best First'.
Re^19: does threads (still) memleak?
by BrowserUk (Patriarch) on Nov 28, 2008 at 15:50 UTC
    Maybe the problem arises when trying to cycle through all keys of the hash at the exact same time its contents are being updated:

    That makes a certain amount of sense in the version above with the sleeps removed. It's quite easy to see how (without locking) that the hash iterator could be corrupted. But that does not explain the earlier versions where you have sleeps that effectively orchestrate serialisation of the writing and reading of the hash.

    Yes, you should be using locking. But for locking to be the fix for the original (with sleeps) version of the code, it would mean that iterating a 2 key hash would have to take longer than not just one timeslice (in order for the other thread to have a need to lock), but longer than 1 second. And that's just silly.

    A quick test shows that Perl can iterate a two-key shared hash 10 million times in just over 4 seconds: 

    use threads; 
use threads::shared;
use Time::HiRes qw[ time ];

my $h:shared = ( 1..4 ); 
print time; 
for ( 1..1e7 ) { 
    1 while each %h;
}
print time;

1227886927.80338
1227886932.10025

    [download]

    Even on a multicore box and with Data::Dumper in the mix, there's no way that the sleeping version of the code you posted could be concurrently modifying and iterating that hash.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      Are you saying 726584 suggests a bug but 726616 doesn't? I mean in the first example the sleep is inside the sub, so executed almost at the same time for both threads. I don't see the difference.
[reply]

        I'm saying that in the following code: 

        #!/usr/bin/perl
use warnings;
use strict;
use Time::HiRes qw[ time ];
use threads;
use threads::shared;
use Data::Dumper;

my $href = &share({});

sub sub1 {
  sleep 1;
  print threads->tid(), ' : ',  time, $/;
  print Dumper $href;
  print threads->tid(), ' : ',  time, $/;
}

$href->{a} = &share({});

print threads->tid(), ' : ',  time, $/;
my $th1 = threads->new('sub1');
print threads->tid(), ' : ',  time, $/;
$href->{b} = &share({});
print threads->tid(), ' : ',  time, $/;
my $th2 = threads->new('sub1');
print threads->tid(), ' : ',  time, $/;

$th1->join();
$th2->join();

        [download]

        That between the first thread starting its 1 second sleep, and the second thread starting its one second sleep, the main thread has to start that second thread. So the differential between thread 1 iterating the shared hash, and thread 2 iterating the shared hash, even on a system that can run all 3 threads immediately and concurrently, has to be at least the time taken to spawn a new thread.

        And given I know some of what is involved in spawning a new thread, I find it totally improbable that it would take the first thread longer to iterate the two-key shared hash than it would take the main thread to request the OS to create a new thread, and then go through the process of cloning the spawning threads environment into that new thread.

        And that is what it would take for those two threads to be iterating the shared hash concurrently. And that just doesn't add up.

        The timing produced by the code above show that with the two threads dumping the hash with a time differential of at minimum, one thread spawn&clone operation apart, the possibility that first thread is still in the progress of iterating the hash when the second thread starts it very unlikely. There is a 3-orders of magnitude differential involved. 

        C:\test>junk2
0 : 1227891440.24088 } 
0 : 1227891440.26269 } 22 milliseconds to start a thread
0 : 1227891440.26288
0 : 1227891440.28761 } 24 milliseconds to start a thread

1 : 1227891441.27213  }
$VAR1 = {
          'a' => {},
          'b' => {}
        };
1 : 1227891441.27275  } 62 microseconds to dump the hash

2 : 1227891441.28775  }
$VAR1 = {
          'a' => {},
          'b' => {}
        };
2 : 1227891441.28835  } 60 microseconds to dump the hash

        [download]
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom