in reply to Re^8: Win32::Process output
in thread Win32::Process output

  1. What is the purpose of $semSTDOUT ?

    If multiple threads print to a filehandle at the same time, their output can become interleaved. Ie. bits of one line get mixed into another line. Like two people talking at the same time.

    The shared variable: $semSTDOUT simply acts as a semaphore to prevent this. By locking that variable before printing, it ensures that each line of print is output whole.

    If you've read "Lord of the Flies", the variable is acting like the conche shell.

  2. What does threads->tid do?

    It returns the thread id of the "current" thread.

    The reason it always returns 0 in the example, is because tprint() is only ever called from thread 0--the main thread. But if you add some trace to the other threads, each threads output will be prefixed by its thread id which is useful for debugging.

  3. Does the queue itself maintain the integrity (i.e. 3 worker threads ask for the next name from the queue, does the queue get locked for each call to prevent duplication?)

    Yes. Thread::Queue takes care of all the required locking. Each $Q->dequeue() will return the next value from the queue. Once read, it is removed and no other thread will ever know it was ever there.

  4. Is the outer 'master thread' that feeds the names from the file into the queue strictly necessary?

    If the work Q feeding code was not in a separate thread from the results Q reading code, then you would not start to see the results until all the work items had been queued and processed.

    You could feed the queue in a single burst, but if the list is large, that would consume a large amount of memory. Better to keep the queue size small by feeding just enough to keep the workers busy.

    You could try to multiplex the feeding of the work Q and the reading of the results Q. But that just gets messy and creates potential for deadlocks.

    Rather than trying to do two different things a the same time in one thread--with all the synchronisation problems that creates--better to start another thread and let each thread concentrate on doing one thing simply.

    That's the purpose of threading.

Thanks, this seems to be a good shell for what I need to do (though I'll probably wear out the Perl book figuring it out).

NP. If you have any further questions, do ask them. It is far easier to answer your questions, than to try and predict every question you might ask, and clutter the code with long rambling comments attempting to answer them.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^10: Win32::Process output
by tawnos (Novice) on Nov 14, 2010 at 17:11 UTC

    OK so I didn't understand as much as I had hoped, and tried to write something much simpler. Even that didn't work, and I'm not sure what I'm doing wrong. I figured I would remove the whole 'terminating a process' part from the equation and try to add that in later and just try to make a threaded ping script

    use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Net::Ping;

use constant NUMBER_OF_THREADS => 20;
use constant TIMEOUT => 4;
use constant INPUT_FILE_NAME => "slist.txt";

our %results : shared;

sub pingworker
   {
   my ($Qin, $timeout) = @_;
   my $response;
   while (my $server = $Qin->dequeue)
      {
      $response = Net::Ping->new()->ping($server);
      lock(%results);
      $results{$server} = $response ? "Alive" : "Timed Out";
      }
   }
 
my $Qwork = Thread::Queue->new;

my $thread;
for (1 .. NUMBER_OF_THREADS)
   {
   $thread = threads->new(\&pingworker, $Qwork, TIMEOUT);
   $thread->detach;
   }

open(SLIST, INPUT_FILE_NAME) || die "Cannot open server list: $!\n";
while (<SLIST>)
   {
   chomp;
   $Qwork->enqueue($_) if /[A-Za-z0-9\.\-_]+/;
   sleep 0.5 while $Qwork->pending > NUMBER_OF_THREADS;
   }
close(SLIST);

$Qwork->enqueue( (undef) x NUMBER_OF_THREADS);

sleep 0.5 while $Qwork->pending > 0;

foreach my $server (sort keys %results)
   {
   print $server . "," . $results{$server} . "\n";
   }

    [download]

    So this actually seems to work (and I can understand the syntax) except for the fact that the results are completely wrong (its reporting most of the targets as non-responsive when they are). When i've tried to do output from the threads by locking $semSTDOUT it appears that the first time through it gets a real response from ping and after that or (seemingly) sometimes the second time through it starts reporting everything as nonresponsive.

    I still don't understand the purpose of the output queue (as opposed to just stuffing it into a big array or something) in the original example, but that doesn't seem to be my big problem. I would guess thats for the case where the results need to be output while the script is running (e.g. if it were too large to store in memory?)

[reply]
[d/l]

In Section Seekers of Perl Wisdom