in reply to Perl Threads

The problem is that you are creating all your threads; then joining them. Although you are limiting the number of running threads; because you aren't joining them; they are all hanging around in their non-running, joinable state; and that it what is consuming your memory.

Given that you aren't retrieving anything from the threads when you join them, the simple solution is to avoid having to join them, by detaching them at the point of creation.

Instead of this: $active_threads{ @row } = threads->create ( \&deleteDevice, @row );

Do this: threads->create( \&deleteDevice, @row )->detach;

That way the threads will clean themselves up automatically on termination.

However, there is a problem with this; namely that once your main loop creating the threads finished, you can no longer use: 

foreach( threads->list() ) { $_->join(); }

[download]

to ensure that the main thread doesn't end before the worker threads complete; and unfortunately, the authors of threads didn't provide a mechanism for waiting until all detached threads complete; nor even a way of checking if any detached threads currently exist.

There are several approaches to dealing with this; some more complicated than others and most have their own sets of problems. The simplest way I know is to use my count-them-out-and-count-them-back method:

use threads::shared; ## at the top
...
my $running : shared = 0; ## before the main loop

...

sleep 1 while $running; ## after the main loop in-place of the join lo
+op

sub deleteDevice($) {
    ++$running;
    .... # as now
    --$running;
}

[download]

Add that lot together and your memory problems should go away.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.
I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!

Replies are listed 'Best First'.
Re^2: Perl Threads ( main thread is detached )
by Anonymous Monk on Aug 13, 2015 at 23:40 UTC
    Hmm, main thread is detached? 
    $ perl -Mthreads -e " threads->detach "
Thread already detached at -e line 1.

$ perl -Mthreads -e " threads->self->join "
Cannot join a detached thread at -e line 1.

    [download]
[reply]
[d/l]
      Hmm, main thread is detached?

      How could it be otherwise? What part of the process could join the main thread?

      The main thread, is the process. Its return value can only be retrieved by another process; in which context thread->join doesn't make any sense.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
      I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
[reply]
Re^2: Perl Threads
by Monkless (Acolyte) on Aug 14, 2015 at 17:10 UTC

    So taking your suggestions above, I have updated my scipt as seen below and this does seem to resolve my memory issue - MUCH APPRECIATED! 

    my $running : shared = 0;

###THREADED###
##Parse Fetch and execute DELETE
while (@row = $sth->fetchrow_array )
{
    ##Create and Run threads
    ##Limit running threads
    sleep 2 while $running >= $maxThreads;
    threads->create (\&deleteDevice, @row)->detach;
}

##Thread cleanup
sleep 1 while $running > 0;



sub deleteDevice($)
{
    ++$running;
    my $deviceSN = shift;
    my $full_url = "https://$api_creds\@$cwm_url$deviceSN";
    &debug("CMD: $full_url");

    #my $threadCount = scalar threads->list(threads::running);
    print "Current Thread Count: $running\r";

    # Get current subscriber info
    my $ua = LWP::UserAgent->new(
            ssl_opts => { verify_hostname => 0, SSL_verify_mode => 0x0
+0 }
    );
    $ua->agent("Device Deletion");
    
    my $req = HTTP::Request->new(DELETE => $full_url);
    $req->content_type('application/x-www-form-urlencoded');

    my $res = $ua->request($req);
    if($res->is_success)
    {
        &logDeviceDetails("$deviceSN -- Delete Successful");
                        
    }
    else
    {
        &logDeviceDetails("$deviceSN -- Delete Failed: " . $res->statu
+s_line . " - " . $res->content)

    }
    --$running;

}

    [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom