exodist has asked for the wisdom of the Perl Monks concerning the following question:

I am writing a threaded program, it has to be threaded, no way around this, if you are only gonna suggest I do not do this threaded pleases top reading now and do not reply. I have one thread that has a buffer, it puts the request into a que, another thread picks up the request from the que, and fills the buffer (part of the request). The thread that made the request needs to wait until he buffer is full and then continue. Here is the code I was thinking would do the trick, I cannot test it at the moment, I am wondering if it will work or if there is a better way.
my $Buffer; #The buffer that needs to be filled. push(@Que, \$Buffer); #Put the buffer into the que, this is a line I p +ut in for simplicty, the actual request is a lot more complicated. while (1) #Infinite loop { threads->yield(); #Yield each iteration lock($Buffer); #Lock the buffer, this way if data is still being p +laced intot eh buffer in the other thread it will be locked there and + we cannot get data early. will it be unlocked for the next iteration +? or is the loop not considered leaving it's own {} per iteration? last if ($Buffer); #If the buffer is filled continue. }

--------------------------------------

I would rather take 30 minutes to re-invent the wheel then take 30 days to learn how to use someone else's. Before pestering me about my re-invention prepare to defend yourself with a way of learning how to use the wheel in less time than it takes for me to make one, one that I might add is specialized to my specific task!

Replies are listed 'Best First'.
Re: need a thread to wait until a buffer is full.
by Trizor (Pilgrim) on Jul 24, 2007 at 21:54 UTC

    I don't think you're understanding the semantics of lock. When you call lock on a locked variable the call blocks until the lock is freed. Every loop iteration is its own lexical scope, so when you fall off the end locks are freed. Your comments say that the other thread won't unlock the $Buffer until after its full, so the last if ($Buffer); to continue on your program is unnecessary unless your semantics differ from what you've told us. To ensure timely lock freeing you can use bare blocks:

    #initialize... { lock($Buffer); # We've got the lock # Do stuff } # Fall off the end of our enclosing scope

    Also for your queue have you considered using a Thread::Queue if $Buffer is a string buffer this will be easier than dealing with making sure locks are controlled properly. If the filling thread fills it before relinquishing the lock then instead of mucking with the locks push the full buffer into the queue and then the processor of these buffers just dequeues the full buffers and acts on them. The dequeue method simply waits for data, and you can have multiple threads waiting on a Queue so your processing thread (if it doesn't require atomic output or something) can have multiple (number of cores + 1) instances processing.

Re: need a thread to wait until a buffer is full.
by BrowserUk (Patriarch) on Jul 24, 2007 at 23:04 UTC

    Something like this?

    #! perl -slw use strict; use threads; use threads::shared; use Thread::Queue; $|++; sub filler { my( $Q ) = @_; while( my $ref = $Q->dequeue ) { lock $$ref; $$ref .= 'X'x10 while sleep 1 and length $$ref < 30; cond_signal $$ref; } } my $Q = new Thread::Queue; my $buffer :shared; my $filler = threads->create( \&filler, $Q ); for( 1 .. 5 ) { lock $buffer; $Q->enqueue( \$buffer ); cond_wait( $buffer ); print $buffer; $buffer = ''; } $Q->enqueue( undef ); ## kill thread $filler->join;

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      The things you put in a Thread::Queue don't need to be shared. Just the Thread::Queue itself.
        he things you put in a Thread::Queue don't need to be shared. Just the Thread::Queue itself.

        They do if you want access them from more than one thread. Note that I am passing a reference to the shared buffer via the Queue, not the buffer itself.

        If you can see a better way of coding this, please post your code.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: need a thread to wait until a buffer is full.
by archfool (Monk) on Jul 24, 2007 at 21:42 UTC
    Have you looked into Thread::Queue? I believe it handles the mutexes and locking for you. It can be :shared between threads. You ->enqueue your data in one thread, and ->dequeue (blocking) in the other thread. (Or ->dequeue_nb for non-blocking).
      Ok, here is a better way to ask my question:

      I have 2 threads, one creates shared variable that acts as a buffer, it puts a reference to the buffer in an array, then it waits until a second thread puts something into the buffer before continuing.

      The second thread will pop/shift stuff out of the array every once in a while, fill the buffers according to some request that is sent with them.

      I need to make sure the first thread waits until there is something in the buffer, I do not know when the buffer is going to be filled, and I have no idea how much data will be in the buffer when it is done, but it will be binary data, so it can be ANYTHING.

      My idea was to create a loop in the first thread where it does the following (not necessarily in this order)
      • Lock the buffer, this way if it is being written to we wait instead of assuming the other thread has finished with it.
      • If the buffer contains data continue
      • If the buffer is still empty unlock it I know to unlock it it needs to fall out of scope, ie end of {} block... I am not sure if a loop iteration counts as ending the scope.
      • Yield to other thread so it can get to the buffer.

      this is a loop, I figure the yield should come first since the first iteration will almost certainly be too soon. then it locks it, once it obtains the lock it checks it, if it is empty we reach the end of the loop. this is the part that worries me: at the end of the loop, the loop starts over, I am not sure if a loop iterating a second time will release the lock or not, I am sure it is, but it would be hard to debug if not.
      so the code I get to do this is roughly this:
      while (1) #Infinite loop { threads->yield(); #Yield each iteration lock($Buffer); #Lock buffer to prevent taking a partialy filled bu +ffer. last if ($Buffer); #If the buffer is filled continue. }

      --------------------------------------

      I would rather take 30 minutes to re-invent the wheel then take 30 days to learn how to use someone else's. Before pestering me about my re-invention prepare to defend yourself with a way of learning how to use the wheel in less time than it takes for me to make one, one that I might add is specialized to my specific task!