Thread::Queue vs Thread::Queue::Any

gulden has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

Should we use Thread::Queue::Any instead of Thread::Queue?

I've made a simple script to test them (see below), and the results were:

------------------------------------
TEST QUEUE:51(seconds)
------------------------------------

------------------------------------
TEST QUEUE ANY:27(seconds)
------------------------------------
[download]

I've detected a lot more of LOCKs when running Thread:Queue rather than with Thread::Queue::Any.

#!/opt/coolstack/bin/perl

use strict;

use threads ('yield',
                 'stack_size' => 64*4096,
                 'exit' => 'threads_only',
                 'stringify');

use Thread::Queue;
use Thread::Queue::Any;


my $nloaders = 16;
my $nworkers = 16;
#---------------------------------------------------------------

my $queue = Thread::Queue->new();
my $queue_any = Thread::Queue::Any->new();


my $st = time;
#----------------
# TEST QUEUE
#----------------
my @thrs_loaders;
push @thrs_loaders, threads->create(\&load_hash, 0) for (1..$nloaders)
+;

my @thrs_workers;
push @thrs_workers, threads->create(\&process_hash, 0) for (1..$nworke
+rs);
$_->join for @thrs_loaders;
$queue->enqueue({'STOP' => 1}) for (1..$nworkers);
$_->join for @thrs_workers;


print "------------------------------------\n";
print "TEST QUEUE:" . (time - $st) . "(seconds)\n";
print "------------------------------------\n";

print "Enter an key to continue...\n";
<STDIN>;
$st = time;
#----------------
# TEST QUEUE ANY
#----------------
@thrs_loaders = ();
push @thrs_loaders, threads->create(\&load_hash, 1) for (1..$nloaders)
+;


@thrs_workers = ();
push @thrs_workers, threads->create(\&process_hash, 1) for (1..$nworke
+rs);
$_->join for @thrs_loaders;
my %hash = ('STOP' => 1);                      # Updated after Browser
+Uk Comments
$queue_any->enqueue(%hash) for (1..$nworkers); #
$_->join for @thrs_workers;

print "------------------------------------\n";
print "TEST QUEUE ANY:" . (time - $st) . "(seconds)\n";
print "------------------------------------\n";


exit;

#---------------------------------------------------------------------
sub load_hash{
  my $test_queue_any = shift || 0;
  my $id = threads->tid();
  my $tmp;

  print "START LOAD $id\n";

  for my $i(1..10000){
    my %rec = ('ola'  => '1', 'ola1' => '2' );
    if($test_queue_any){
      $queue_any->enqueue(%rec);
    }else{
      $queue->enqueue(\%rec);
    }
  }

  print "STOP LOAD $id\n";
}

#---------------------------------------------------------------------
sub process_hash{
  my $test_queue_any = shift || 0;
  my $id = threads->tid();
  my $tmp;

  print "START WORKER $id\n";

  if($test_queue_any){
    while(my %hash = $queue_any->dequeue()){
      last if exists $hash{'STOP'};
    }
  }else{
    while( my $ref = $queue->dequeue() ){
      last if exists $ref->{'STOP'};
    }
  }

  print "STOP WORKER $id\n";
}
[download]

Comment on Thread::Queue vs Thread::Queue::Any Select or Download Code

Replies are listed 'Best First'.
Re: Thread::Queue vs Thread::Queue::Any by BrowserUk (Patriarch) on Jul 14, 2009 at 14:38 UTC
I'm very surprised at your results! T::Q::A is a subcless of T:Q, so it does everything that TQ does, and some extra stuff on top, so how can it be quicker? Especially as the 'extra' it does is to serialise the data passed into a mod//Storable`::Freeze` bytestring when enqueuing and unfreezing when dequeing. Neither of which is know to a an especially fast operation. What puzzles me even more is that your benchmark is broken: `if($test_queue_any){ $queue_any->enqueue(%rec); }else{ $queue->enqueue(\%rec); }` [download] For TQA, you're passing the hash as a flattened list; `$queue_any->enqueue(%rec);` which internally becomes the array `@_`; which is then converted to a single (freeze) string; `shift->SUPER::enqueue( Storable::freeze( \@_ ) );` and push onto the TQ queue proper. At the other end a single string is read off the queue; thawed back to a flat list and return to the caller; `@{Storable::thaw( shift->SUPER::dequeue )};` where you reconstruct a hash from the list: `while(my %hash = $queue_any->dequeue()){` For TQ, you queue a scalar (reference) directly to the queue, and the return that reference to the caller. So both exchange a single scalar between the threads, but one does it directly; the other indirects through a subclass and flattens, freezes, thaws and reconstructs the hash on the way through. How could the latter possibly be faster? How could it possibly show fewer lock states--since the exact same underlying code is being used in both cases? So I ran you code and got: C:\test>TQ-TQA-b.pl START LOAD 1 START LOAD 2 ... START LOAD 15 START LOAD 16 START WORKER 17 START WORKER 18 ... START WORKER 31 START WORKER 32 STOP LOAD 6 STOP LOAD 1 ... STOP LOAD 12 STOP LOAD 16 STOP WORKER 23 STOP WORKER 19 ... STOP WORKER 20 STOP WORKER 26 ------------------------------------ TEST QUEUE:15(seconds) ------------------------------------ Enter an key to continue... START LOAD 33 START LOAD 34 ... START LOAD 47 START LOAD 48 START WORKER 49 START WORKER 50 ... START WORKER 63 START WORKER 64 STOP LOAD 33 STOP LOAD 35 ... STOP LOAD 48 Odd number of elements in hash assignment at C:\test\TQ-TQA-b.pl line +80, <STDIN> line 1. STOP WORKER 58 Odd number of elements in hash assignment at C:\test\TQ-TQA-b.pl line +80, <STDIN> line 1. Odd number of elements in hash assignment at C:\test\TQ-TQA-b.pl line +80, <STDIN> line 1. ... Odd number of elements in hash assignment at C:\test\TQ-TQA-b.pl line +80, <STDIN> line 1. STOP WORKER 53 STOP WORKER 63 ------------------------------------ TEST QUEUE ANY:20(seconds) ------------------------------------ [download] Besides the warnings, which are essentially trivial compared to the other brokeness of the benchmark--more a sign of your lack of understanding of hashes than a serious problem--I get TQA taking 33% longer than TQ; which gels with my expectations! Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP PCW	[reply] [d/l] [select]
Re^2: Thread::Queue vs Thread::Queue::Any by gulden (Monk) on Jul 14, 2009 at 21:48 UTC
ŤHow could the latter possibly be faster? How could it possibly show fewer lock states--since the exact same underlying code is being used in both cases?ť You got it!! The answer for your questions is what I want!!	[reply]
Re^3: Thread::Queue vs Thread::Queue::Any by BrowserUk (Patriarch) on Jul 14, 2009 at 21:55 UTC
It can't! Therefore your measurements are wrong!	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.
A reply falls below the community's threshold of quality. You may see it by logging in.
A reply falls below the community's threshold of quality. You may see it by logging in.