memory leak with Thread::Queue ?

Random_Walk has asked for the wisdom of the Perl Monks concerning the following question:

Morning all

I am having problems with my multithreaded tcp listener with IO::Socket. It is leaking memory even after applying the good advice from BrowserUk. I suspect it may be a problem with the Thread::Queue. Here is a simple script that shows the problem (unixy 'cos of ps command, strip that out and use taskmangler to see it on windows).

#!/usr/bin/perl
use strict;
use warnings;
use threads;
use Thread::Queue;
my $queue = Thread::Queue -> new;

threads -> create ("popper", $queue) -> detach for (1..10);

my $message="this is a line of test around the right size";

while (1) {
    $queue -> enqueue ($message) for (1..100);
    sleep 1 while $queue -> pending;
    print $/,`ps -eo vsize,args | grep $0 | grep -v grep`;
}

sub popper {
    print "started a popper\n";
    my $queue = shift;
    while ($queue -> dequeue) {
        print ".";
    }
}
[download]

Here is a snatch of output ...

 6300 /usr/bin/perl ./threadtest.pl
......................................................................
+..............................
 6316 /usr/bin/perl ./threadtest.pl
......................................................................
+..............................
 6340 /usr/bin/perl ./threadtest.pl
......................................................................
+..............................
 6364 /usr/bin/perl ./threadtest.pl
......................................................................
+..............................
 6408 /usr/bin/perl ./threadtest.pl
......................................................................
+..............................
[download]

Is this because a variable on the queue is being made shared between all threads and never going out of scope ? If so is this fixable or an imutable feature of thread queues ?

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Comment on memory leak with Thread::Queue ? Select or Download Code

Replies are listed 'Best First'.
Re: memory leak with Thread::Queue ? by BrowserUk (Patriarch) on May 16, 2006 at 10:04 UTC
Your code (slightly modified for win32), doesn't appear to leak under AS811/5.8.6 or AS817/5.8.8. #!/usr/bin/perl use strict; use warnings; use threads; use Thread::Queue; my $queue = Thread::Queue -> new; threads -> create ("popper", $queue) -> detach for (1..10); my $message="this is a line of test around the right size"; while (1) { $queue -> enqueue ($message) for (1..100); sleep 1 while $queue -> pending; system qq[tasklist /nh /fi "PID eq $$"]; } sub popper { print "started a popper\n"; my $queue = shift; while ($queue -> dequeue) { print "."; } } __END__ C:\test>junk5 started a popper started a popper started a popper started a popper started a popper started a popper started a popper started a popper started a popper started a popper perl.exe 3128 0 6,936 +K perl.exe 3128 0 6,944 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re^2: memory leak with Thread::Queue ? by Anonymous Monk on Sep 18, 2009 at 22:04 UTC
This problem is present on RHEL5 running 5.10.0, and RHEL5 running 5.10.1. It is NOT present on RHEL4 running 5.10.0. I don't understand why.	[reply]
Re^3: memory leak with Thread::Queue ? by BrowserUk (Patriarch) on Sep 18, 2009 at 22:24 UTC
I'd suggest you re-post this under dave_the_m's post above (or take it directly to p5p), as he understands the problem and is best place to suggest how to fix it. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP PCW It is as I've been saying!(Audio until 20090817)	[reply]
Re: memory leak with Thread::Queue ? by dave_the_m (Monsignor) on May 16, 2006 at 09:05 UTC
What version of perl are you using? I believe that leak was fixed in 5.8.1 Dave.	[reply]
Re^2: memory leak with Thread::Queue ? by Random_Walk (Prior) on May 16, 2006 at 13:14 UTC
Hi Dave_the_m I think you have hit the nail on the head >perl -v This is perl, v5.8.0 built for aix-thread-multi Copyright 1987-2002, Larry Wall Perl may be copied only under the terms of either the Artistic License + or the GNU General Public License, which may be found in the Perl 5 source ki +t. Complete documentation for Perl, including FAQ lists, should be found +on this system using `man perl' or `perldoc perl'. If you have access to + the Internet, point your browser at http://www.perl.com/, the Perl Home Pa +ge. [download] I will test it on some later flavours. Update Sure enough a test on 5.8.7 works fine as well. Version to blame, sorry for the inconvenience Many thanks, R. Pereant, qui ante nos nostra dixerunt!	[reply] [d/l]

Update