in reply to Re^6: Problem in Inter Process Communication
in thread Problem in Inter Process Communication

Also in the model that u have proposed how do i add support for handling multiple queries,i can use the same strategy as i had used before...

Yes. Second and subsequent queries to the same client will just get picked up by the next available worker.

However, if it is important to only issue a single query against a given client at any one time, you'd have to take additional steps to ensure that. If that is the case, speak up and I'll see what I can come up with.

A few comments on your code:

  1. I generally prefer to start the workers, before adding the work items to the queue.

    More habit than for any reason I can remember, but the workers will just block on the queue until there is something to do anyway. And I have vague recollections of it making a difference on at least one occasion.

  2. There is no real benefit to using a hash to store your thread handles.

    An array is simpler and works better.

  3. There is no need to pass an integer to each thread as an identifier.

    You can obtain the process unique thread id using:

    my $tid = threads->self->tid;

    Where threads->self is a class method that returns the thread handle of teh current thread and ->tid is an instance method that return the thread identifier for the invocant thread handle.

  4. undef is an intrinsic value (Not a string as you have it).

    And whilst you were only starting a single thread for now, it better to use the configuration variable $maxnoofThreads to ensure sufficient undefs are stacked to terminate all the threads.

    The construct:

    $q->enqueue( ( undef ) x $maxnoofThreads );

    Says: build a list of $maxnoofThreads undefs and push them onto the queue.

  5. As you only put the handles of the worker threads into the list, there is no need to check whether one of them is the main thread.
  6. There is no need to return unless $workitem.

    Once all the work items have been processed, each thread will dequeue an undef, and the while loop will terminate naturally. and the thread will terminate when it "falls of the end" of the worker sub.

  7. And time you use a shared resource (like printing to the terminal or a file), from multiple threads, you should serialise acccess to that resource.

    Ie. Apply locking to a shared variable.

    On some systems, printing to the screen will be serialised by the OS or runtime, but you cannot rely on that everywhere. Also, if the output is redirected to a file, the automatic serialisation goes away.

Putting that all together, it looks like this:

use strict; use threads; use threads::shared; use Thread::Queue; my $mtxStdOut : shared; my $maxnoofThreads = 1; my @clientList = qw( client1 client2 client3 client4 client5 client6 client7 client8 client9 client10 client11 client12 ); my $q = new Thread::Queue; my @workers = map { threads->new( \&worker, $q ); } 1 .. $maxnoofThreads; $q->enqueue(@clientList); $q->enqueue( ( undef ) x $maxnoofThreads ); $_->join for @workers; sub worker { my( $Q ) = shift; my $tid = threads->self->tid; while( my $workItem = $Q->dequeue ) { { lock $mtxStdOut; print " workitem from thread $tid -->$workItem\n"; } ## Perform query ## Perform comparison ## Perform output/cleanup } }

That's a very lightly customised template that I use for most threaded perl applications. It's flexible enough that it lends itself to many uses and is tested well enough that it just works in most cases.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^8: Problem in Inter Process Communication
by libvenus (Sexton) on Aug 22, 2008 at 03:13 UTC

    One more question on the strategy lets say if i use the boss and worker thread model and make two boss( query and comp ) . The query boss would make a threaded queue and create more workerthreads to work on the query and client.Each workerthread would loop on the clients and after comleting for one client it would dump the output in a file and enqueue the locaiton of the file in a global threaded queue.The comp boss thread has launched x no of workerthreads which are blocking on the global theaded queue.

    Does this model looks feasible+scalable. I know the IO would be increase but is it possible and recommended.

      Yes. It is theoretically feasible. But you have to come bak to the question, what are you gaining (or hoping to gain) by going the route of create threads on the fly?

      There are legitimate reasons for doing this, but it is an expensive process that adds complications, some of which can be extremely difficult to resolve. So before I would consider going this route, I would need to have good reasons for doing so, and on the basis of what you told us so far, I do not see that reason?

      What are your reasons for thinking this is a good idea?

      Also, it would be easier to advise you if you woudl describe more details of the processing you are doing. For example, you say you are querying clients, but what type of clients and what type of queries?

      And the second part of your processing is "comparing"...but comparing what? This is a comaprison:

      if( $x == $y ) { #do A } else { #do B }

      but it certainly wouldn't be worth starting a new thread in order to perform that operation. However, if you are comparing several megabyts of data, it might make sense.

      For better advise, you need to explain more of what you need to do. And for serious consideration of your architectural questions, you need to explain why you are thinking of implementing them that way.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Hi browserUk,

        Thanks for your help !!!

        Well i have been assigned a task to migrate a regression suite from c++ to perl. The reg suite has a task to fire queries to prod and test and then compare the results.The queries are fired to a few app servers. The no of client which must be replaced in each query is around 10K and the no of queries is around 5K.The result of the query fired can contain 10 K records at times.Firing query generally takes much less time than comparison and this is the primay reason y i want to segregate things this way. Each query must be replaced by all the clients and then fired onto the app servers