Hey all, I seem to have a memory leak in my threaded script... I haven't done much thread work so it is probably an oversight on my part, I'd appreciate a look.

This is a simplified version that exhibits the problem, I am trying a classic method of one worker filling a queue with multiple workers reading from the queue and managing the data, my total dataset is over a million records so I am breaking it into 10000 record chunks, but it would seem they are hanging around in memory. This example just makes data forever but clearly shows that the data is sticking around when you watch top.

I am using Perl 5.8.5, threads 1.73, threads::shared 1.29 and Thread::Queue 2.11, all on Linux

use threads;
use threads::shared qw(share);
use Thread::Queue;
use Thread::Semaphore;

my $q : shared       = Thread::Queue->new();
my $loading : shared = 1;
my $threads : shared = 0;
my $thr = threads->create( \&load_queue, $q, \$loading );
$thr->detach();
$|++;

my @threads;
for ( 1 .. 6 ) {
    my $thr = threads->create( \&{type_issues}, $q, \$loading, $thread
+s );
    push @threads, $thr;
}

$_->join() for @threads;

sub load_queue {
    my $q       = shift;
    my $loading = shift;

    my $i = 1;
    my $ary = &share( [] );
    while ( my $v = ["this is a test" x 250] ) {

        #        print "$v->[0]\n";
        push @$ary, $v->[0];
        unless ( $i++ % 10000 ) {
            print "$i\n";
            $q->enqueue($ary);
            $ary = &share( [] );
            while ( $q->pending() > 20 ) {    # try not to eat tooo mu
+ch memory
                 select(undef,undef,undef,.1);
            }
        }
    }
    $q->enqueue($ary) if @$ary;
    $$loading = 0;
}

sub type_issues {
    my ( $q, $loading ) = @_;
    #my $DB = IDC::Data->new( bes => 'webdev' );
    print "Type issues\n";
    my $issues;
    while ( ( $issues = $q->dequeue() ) ) {
        print "Got ".@$issues." - ".$q->pending()."\n";
        select(undef,undef,undef,.1);
    }
}

[download]

                - Ant
                - Some of my best work - (1 2 3)

In reply to Threaded script eating memory by suaveant

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.