failing with Thread::Queue

rastoboy has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, friends!

So just when I think I've got this threading thing figured out...;-)

I'm trying to use Thread::Queue to coordinate some simple activities which need to occur simultaneously. Basically I need to run them in a loop until X happens. Basically I'm running a wget, while monitoring cpu and memory usage.

The examples I've seen for how to use this module are all feeding information into a thread, where the thread then acts on them in real time. I just want it to continue doing something until I say stop. The technique I'm using--queueing a single bit of data and using Thread::Queue->peek() in the thread to see if it's true, seems to work fine when I have a single thread. But if I have more than one thread, only the last thread created seems to be able to use peek():

#!/usr/bin/perl

use strict;
use warnings;

use threads;
use threads::shared;
use Thread::Queue;
my $q = Thread::Queue->new;

my $done : shared;
$done = 0;

my $wget_thread = threads->create( \&wget_sub );

my $cpu_monitor_thread = threads->create( \&monitor_cpu );
my $mem_monitor_thread = threads->create( \&monitor_mem );

$q->enqueue(1);
$wget_thread->join();
$q->dequeue;

my $cpu = $cpu_monitor_thread->join();
my $mem = $mem_monitor_thread->join();

print "cpu usage: $cpu\n";
print "mem usage: $mem\n";

sub wget_sub {
    my $wget = `wget http://10.3.13.4/testfile.dat`;
    $done = 1;
    lock($done);

    print "wget:\n$wget\n";

    return 1;

}

sub monitor_cpu {

    my $cap;
    while ( my $val = $q->peek(0) ) { #this does not get evaluated to 
+true, ever
       $cap .= `bash cpu.sh; sleep 2`;
    }
    return $cap;
}

sub monitor_mem {

    my $cap;
    while ( my $val = $q->peek(0) ) {
       $cap .= `perl mem.pl; sleep 2`;
    }
    return $cap;
}
[download]

In this case, only the monitor_mem loop gets executed. If I comment out the memory monitoring thread creation and joining, the cpu stuff works fine.

What am I doing wrong? Any input would be greatly appreciated!

peek(0) ) { $cap .= `bash cpu.sh; sleep 2`; } return $cap; } sub monitor_mem { my $cap; while ( my $val = $q-

Comment on failing with Thread::Queue Download Code

Replies are listed 'Best First'.
Re: failing with Thread::Queue by BrowserUk (Patriarch) on Mar 22, 2012 at 20:43 UTC
What am I doing wrong? You are trying to use a Thread::Queue as a semapahore. Which is not its purpose. You are using the peek() method of Thread::Queue. This method, along with insert() and extract() should never have been added to a queue module. They create far more problems than they solve. The specific issue with your code is this: `my $q = Thread::Queue->new; ## create a new EMPTY queue my $done : shared; $done = 0; my $wget_thread = threads->create( \&wget_sub ); my $cpu_monitor_thread = threads->create( \&monitor_cpu ); ## Start y +our threads with the queue empty/ my $mem_monitor_thread = threads->create( \&monitor_mem )` [download] When those threads run: sub monitor_cpu { my $cap; while ( my $val = $q->peek(0) ) { #this does not get evaluated to + true, ever $cap .= `bash cpu.sh; sleep 2`; } return $cap; } [download] The queue is empty so they go straight passed the loop and out the other end. If all you want is a way to tell your threads to stop, use a single shared variable: my $done :shared = 0; ... sub monitor_cpu { my $cap; until( $done ) { $cap .= `bash cpu.sh; sleep 2`; } return $cap; } ... my $mon = threads->new( \&monitor_cpu ); ... do other stuff ## To stop the queue(s) $done = 1; $mon->join; [download] Though quite what the purpose of your cpu/mem monitoring is, I do not understand, because you are only going to get the last report returned, which is utterly meaningless as a measure of anything. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l] [select]
Re^2: failing with Thread::Queue by rastoboy (Monk) on Mar 22, 2012 at 21:08 UTC
Omg BrowserUK, thank you so much. I had tried using a simple shared scalar before, but I thought I had to use cond_wait() and locking and friends and I was having a devil of a time. (Actually I had meant to remove those $done lines from the code I posted here for clarity's sake). I tried what you showed and it worked just fine :-) And I now see how what I did could never work. I think the term for that is "having sh*t for brains" on my part. I moved the $q->enqueue(1) to before the creation of the threads, and it worked as expected. I think it's time for a break :-) Just out of curiousity, why are those methods so bad?	[reply]
Re^3: failing with Thread::Queue by BrowserUk (Patriarch) on Mar 22, 2012 at 21:36 UTC
Re-reading your post, I should also comment on: I had tried using a simple shared scalar before, but I thought I had to use cond_wait() and locking and friends Technically, you should use locking on shared vars. You will rarely, if ever, need to use cond_(), and I would suggest you pretend they don't exist until you encounter the situation that you cannot deal with without them. I write a lot of threaded code, and I've only had to use them about 3 or 4 time in total over the past 8 or 9 years. For this specific case, even the need for lock is moot. The shared var is either true or false. When the value is about to change, readers will either see it just before it changes and loop once more, or they will see it just after it changed and terminate. Locking it before changing it; or before reading it; will make no quantifiable difference to the order in which things happen; it is therefore redundant -- in this case!* With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^4: failing with Thread::Queue by rastoboy (Monk) on Mar 22, 2012 at 22:10 UTC
Re^3: failing with Thread::Queue by BrowserUk (Patriarch) on Mar 22, 2012 at 21:22 UTC
Just out of curiousity, why are those methods so bad? Because a queue is a very simple concept -- you put things in one end and take them out the other in the same order -- and the addition of those methods broke that simple model. The fatuous addition of those methods removes all the guarantees associated with the well-proven abstract data type, rendering it a hybrid mess that is impossible to correctly reason about. It turns a queue (back) into an array -- when the whole purpose of the module is to turn an array into a queue! With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]