in reply to Strange memory leak using just threads

As far as I know, this sort of memory leak is very common in Perl when using threads. The only solution that I know I can count on is to reuse threads, and avoid the create-destroy cycle which leaks memory. See Reusable threads demo

I see the small leak also on my 64 bit slackware with the latest Perl 5.12.2. From what I've been able to figure out, the leak comes from Perl's memory management which holds onto memory because there are ref counts there. See does threads (still) memleak? and Memory and garbage collection and Memory leaks and reference counting within Perl. for examples of previous discussion..... its all been talked about before.

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh
  • Comment on Re: Strange memory leak using just threads

Replies are listed 'Best First'.
Re^2: Strange memory leak using just threads
by BrowserUk (Patriarch) on Sep 20, 2010 at 12:23 UTC

    I can run this for as long as I like: 

    #! /usr/bin/perl
use strict;
use warnings;
use threads;

print "threads: ", $threads::VERSION;
print "perl: ", $];
my @child;
my @list =  1 .. 100;

while (1) {
    my @child = map threads->create("test","$_"), @list;
    $_->join for @child;
}

sub test {
    my ($item) = @_;
}

    [download]

    And the memory usage is rock steady. And that has been true for a long time.

    So, what does that tell you?

    Your knowledge is out of date. Your platform--be it the pthreads or memory management or whatever--has an underlying problem that cannot be directly addressed by, nor attributed to, Perl. So stop doing that.

    BTW: Your "simple reusable threads demo", is way overcomplicated. And always was.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      And the memory usage is rock steady. And that has been true for a long time. So, what does that tell you? Your knowledge is out of date. Your platform--be it the pthreads or memory management or whatever--has an underlying problem that cannot be directly addressed by, nor attributed to, Perl. So stop doing that.

      So what does that tell me? First, his script gained memory and your's didn't. I observe the same behavior. So a script written 1 way in Perl may leak, and written another may not. Furthermore, these 2 sample thread scripts are the ultimate in simplicity. Leakage problems become more pronounced when you start adding objects to threaded programs.

      I have the latest Slackware, and before I believe my slackware c libs are faulty, I have to assume Perl's behavior is somehow causing it.

      Since this memory leak problem keeps coming up, all I can do is to mention that thread reuse does prevent the problem.

      But you are more of a thread expert than me, so if you insist that perl threads do not have memory problems, and it is the underlying c library's fault, I call that a cop out. If threads have no problems with ref counts and memleaks, why did you post threads: spawn early to avoid the crush.? Or is that information from you already outdated?

      I'm not really a human, but I play one on earth.
      Old Perl Programmer Haiku ................... flash japh
[reply]
        First, his script gained memory and your's didn't. I observe the same behavior. So a script written 1 way in Perl may leak, and written another may not

        His original doesn't leak here either. All my changes do is run more threads concurrently in an attempt to amplify any problem should it exist. On my system, it doesn't.

        But Vista and Slackware are fundamentally different systems. Hence my attribution to the platform. If the problem was inherent in the (common) Perl sources, it would manifest itself everywhere. It doesn't. So then you have to start looking at the differences.

        See also, (indirectly via 860733), Dave_the_m's comment.

        I'm pretty sure he runs some version of *nix or other. So, maybe it is not the platform, but how Perl is compiled for a given platform.

        Just attributing it to threads isn't very useful.

        Or is that information from you already outdated?

        4 1/2 years, 2 major releases; 7 or 8 subversion releases. Yes. I would think it is fair to say: that is out of date.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      Your platform--be it the pthreads or memory management or whatever--has an underlying problem that cannot be directly addressed by, nor attributed to, Perl.

      Strangely this problem doesn't affect C and Python threaded programs.

[reply]

        Then that perhaps leave "or memory management or whatever".

        Of course, a memory management problem might be attributed to Perl if you are compling your perl with use_perl_malloc.

        See 860733, and (indirectly) dave_the_m comment.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom