in reply to Re^10: does threads (still) memleak?
in thread does threads (still) memleak?

Why do you assign a nicely shared anonymous hash my $href = &share({});, and then immmediately overwrite it with a non-shared one: $href = {};?

What you are doing is assigning a reference to a shared anonymous hash, to a non-shared scalar. You then immediately overwrite it with a reference to a non-shared anonynous hash.

And you do a similar thing again here:

$href->{a} = &share({}); $href->{a} = {};

It's kind of very surprising to me that anything gets shared this way(*), but if simply remove that bewildering practice, then you'll get something approaching the effect you are looking for:

#!/usr/bin/perl use warnings; use strict; use threads; use threads::shared; use Data::Dumper; my $href = &share({}); #$href = {}; sub sub1 { sleep 1; print threads->tid(); print Dumper $href; } $href->{a} = &share({}); #$href->{a} = {}; my $th1 = threads->new('sub1'); $href->{b} = &share({}); my $th2 = threads->new('sub1'); $th1->join(); $th2->join(); __OUTPUTS_ C:\test>junk1 1$VAR1 = { 'a' => {}, 'b' => {} }; 2$VAR1 = { 'a' => {}, 'b' => {} };

That said, I don't understand why you are doing things the way you are doing them. But I realise this is just a snippet, so it might make more sense in your real code. There seems to be a tendancy to use hash refs where a simple hash would do. This is equivalent to the above, far simpler, and more reliable because perl won't let you accidently overwrite the shared hash with a non-shared one as you've been doing with the unshared reference to a shared hash.

#!/usr/bin/perl use warnings; use strict; use threads; use threads::shared; use Data::Dumper; my %hash :shared; sub sub1 { sleep 1; print threads->tid(); print Dumper \%hash; } $hash{a} = &share({}); my $th1 = threads->new('sub1'); $hash{b} = &share({}); my $th2 = threads->new('sub1'); $th1->join(); $th2->join(); __OUTPUTS__ C:\test>junk1 1$VAR1 = { 'a' => {}, 'b' => {} }; 2$VAR1 = { 'a' => {}, 'b' => {} };

(*)It would be interesting to see what the p5p guys make of that, cos it confuses the begebbers outta me!


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^12: does threads (still) memleak?
by faxm0dem (Novice) on Nov 27, 2008 at 16:52 UTC

    Because I stripped down the part of my code to write a test case and kept the stupid bits that I came up with some time ago to avoid "Invalid value for shared scalar" messages without understanding why they were appearing?

    Thanks a bunch I now understand my error.

    For the record, I was assigning a thread object to an element of this shared hash, and using the above "strange overwriting technique" this somehow avoided the error message. Now I'm creating a "my" object reference, share it and assign it only then to the element of the shared hash.

    Now this seems very complicated and I reckon I must have got lost somehow on the road. It's a bad idea to write production code whilst using modules for the first time, discovering design issues, circling back, discovering other issues... I think I'll start over ;-)

    Now may I use your wisdom to get a hint on how you would design the code? Here's the big picture:

    I'm simply trying to run multiple independent threads, each of which should periodically store some time-series numbers. Then a special thread would periodically take these numbers and do something with them (send them somewhere).

    What I'm doing now is start the N threads plus the sending thread. Each thread handles its own infinite loop, and with its own interval. It stores its computed numbers in this shared hash. Now when the sending thread wakes up, it locks this hash, copies it to another, empties it and unlocks it. Then it can do whatever it likes with the copy

      Here's the big picture: ... periodically store some time-series numbers ... and with its own interval

      As overviews go, that's a little like looking at the moon through the wrong end of a telescope. You kinds know what you're looking at, but the details decidedly fuzzy :)

      A few things that would clarify:

      • How is the shared hash keyed?

        Ie. Does seach thread write to a separate key (or separate set of keys)?

      • What does "time-series" numbers" mean?

        An example would be good.

      • When the sending thread wakes up to take it copy, does it just take whatever is available, or should it only take a "complete set"?

        What I'm getting at here is that threads are non-determanistic, so there is no guarentee that all your generating threads will have been given a timeslice by the scheduler between timeslices of your sending thread. If the time interval (controlled by a sleep?) is sufficiently long, the probability is that they will all get a timeslice in the interval, but it is unwise to rely upon such probabilities if the requirement is that the sending thread should always gather a complete set.

        So, I need to know if that is a requirement? And if it is, what steps are you currently taking to ensure that?

      I'd need a far clearer picture of the task before I'd be prepared to offer a design for it's solution.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        • How is the shared hash keyed?

          Each thread feeds a seperate key e.g. $h->{thX}

        • What does "time-series" numbers" mean?
          $h->{th1}->[[ts1,d11,...,d1N],...,[tsM,dM1,...,dMN]] $h->{th2}->[[ts1',d11',...,d1K'],...,[tsL',dL1',...,dLK']]

          each thread has its own independant [tsI,[dIJ]] values

          tsI are timestamps

          dIJ are data (i.e. temperature, cpu cycles, memory usage...)

        • When the sending thread wakes up to take it copy, does it just take whatever is available, or should it only take a "complete set"?

          It takes whatever is ready, and each [tsI,dI1,...,dIN] should be complete.

        As for the timeslices, each thread handles its own, and yes they are pretty long - at least 15 seconds.
Re^12: does threads (still) memleak?
by faxm0dem (Novice) on Nov 28, 2008 at 13:02 UTC
    Regarding the corrected version of the code, I get different results randomly:
    $ ~/perl/recycleshare3 2$VAR1 = { 'a' => {} }; 1$VAR1 = { 'a' => {}, 'b' => {}, 'a' => {}, 'b' => {} }; $ ~/perl/recycleshare3 1$VAR1 = { 'a' => {}, 'b' => {} }; 2$VAR1 = { 'a' => {}, 'b' => {} };

      Sorry, but I would have to say that the first version of the output you've posted above is impossible. It suggests that you have managed to create a hash with duplicate keys--and that is impossible!

      You're going to have to post the code that you're using, along with the versions of Perl/threads/threads::shared. Nether version of the 'corrected' code I posted could ever produce that first output barring some catastrophic corruption.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        The code I used is pasted here:
        #!/usr/bin/perl use warnings; use strict; use threads; use threads::shared; use Data::Dumper; use Config; printf "perl: %s\n", $Config{version}; printf "gcc: %s\n", $Config{gccversion}; printf "threads: %s\n", $threads::VERSION; printf "threads::shared: %s\n", $threads::shared::VERSION; my %hash :shared; sub sub1 { sleep 1; print STDERR threads->tid(); print STDERR Dumper \%hash; } $hash{a} = &share({}); my $th1 = threads->new('sub1'); $hash{b} = &share({}); my $th2 = threads->new('sub1'); $th1->join(); $th2->join();
        I just reproduced the result:
        $ ~/perl/recycleshare3 perl: 5.10.0 gcc: 4.3.2 threads: 1.67 threads::shared: 1.14 1$VAR1 = { 'a' => {}, 'b' => {} }; 2$VAR1 = { 'a' => {}, 'b' => {} }; $ ~/perl/recycleshare3 perl: 5.10.0 gcc: 4.3.2 threads: 1.67 threads::shared: 1.14 2$VAR1 = { 'a' => {} }; 1$VAR1 = { 'a' => {}, 'b' => {}, 'a' => {}, 'b' => {} };