Hi monks, I had a question about implementing an output queue from multiple threads, using Thread::Queue. It's "working" without warnings or errors, but the memory usage gets way out of control, and it's also a lot slower than using locks and shared variables (which I assume it cause I screwed something up). I understand the memory usage would go up if the output queue was getting more than it could handle. In any case I can't seem to find many examples so any advice would be appreciated.

The reason I wanted to use an output queue is that the worker queue is processing very large files on different computers (so it made sense to use threads). The output needs to be serialized. Previously I used a lock as I said. I heard that using an output queue would be better design. So I want to pass the output from the worker threads to the output queue and print the data there.

sub runSearch {
  my $outputQueue = Thread::Queue->new();
  my $outputThread = threads->create(\&outputSub,$outputQueue);

  my $workerQueue = Thread::Queue->new();
  my @workerThreads = map threads->create(\&workerSub,$workerQueue,$ou
+tputQueue), 1..$numThreads;

  $workerQueue->engueue($_) foreach @computer;
  $workerQueue->enqueue(undef) for 1..$numThreads;
  $outputQueue->enqueue(undef);

  $_->join for @workerThreads;
  $outputThread->join;
}

sub workerSub {
  my ($workerQueue,$outputQueue) = @_;
  while (my $computer = $workerQueue->dequeue()) {
    ## SSH command retrieving grep results and processing
    $outputQueue->enqueue($results);
  }
}

sub outputSub {
  my $outputQueue = $_;
  while (my $packet = $outputQueue->dequeue()) {
    print $results;
  }
}
[download]