Is there a way of using Thread::Semaphore in this way, or an alternative semaphore module that can be used for this purpose?

You don't want a semaphore.

The simplest mechanism is:

wbile( my $workitem = sourceOfWork() ) {
    sleep 1 while $Q->pending > MAXQ;
    $Q->enqueue( $workitem );
}
[download]

If you are concerned that the queue might empty during that 1 second, use Time::HiRes::usleep().

If you really insist on not polling the queuesize, then use a condition vatiable:

my $cond :shared;

...
wbile( my $workitem = sourceOfWork() ) {
    cond_wait( $cond );
    $Q->enqueue( $workitem );
}


## somewhere else ...
...
cond_signal( $cond ) if $Q->pending < MAXQ;
...
[download]

But really, the latter involves more work and is less effective.

You don't want a semaphore.

The simplest mechanism is:

while( my $workitem = sourceOfWork() ) {
    sleep 1 while $Q->pending > MAXQ;
    $Q->enqueue( $workitem );
}
[download]

Thanks, that looks to be a much more sensible approach compared with what I was doing with semaphores.

That should teach me to read algorithoms online that are intened for other languages or problem domains where near realtime responses are required.

The best way (aka not polling) would be that the worker thread will stop itself if the queue is full, and will be restarted by a client thread by a signal.

take a look at thread signalling chapter in threads documentation

Why not use a Thread::Queue to distribute the work to your worker threads? The workers will block when the queue is empty and get woken up when there is work to be done.

I am doing that, the work finding thread is putting the work it finds into a Thread::Queue object.

The reason I want to use a semaphore as well is because the work finder runs much faster than the workers, but is quite expensive in terms of resource consumption.

If there was not a mechanism to pause the work finder when there is enough work in the queue, it would build a queue of millions of items and waste a lot of memory and other resources in the process.

I agree with BrowserUK’s sentiment that a simple polling loop (with a substantial sleep() delay between each iteration, and an appropriately long or short interval) is the simplest approach in this case.   If, in actual practice, a simple poll keeps the work-to-do list reasonably full most of the time, “you’re done.”   If the feeder process “wastes” a few dozen microseconds every thousand, and in the process it avoids a more complicated timing mechanism ... so what ... it’s a “waste” that you can afford.   Even if, every now and then, “the well runs dry,” but the total waste is not a significant delay to the business operation that it supports ... you can afford the simplicity.

package Thread::Semaphore::Boolean;

use threads::shared;

use parent 'Thread::Semaphore';

# Create a new 'boolean' semaphore
sub new {
    my $class = shift;
    return $class->SUPER::new(1);
}

# Set semaphore's count to 1
sub up {
    my $sema = shift;
    lock($$sema);
    $$sema = 1;
    cond_broadcast($$sema);
}

1;
[download]

This is a very bad idea.

A counting semaphore is fundamentally the wrong mechanism(*) for watermarking a queue; so futzing with the API of a counting semaphore to allow it to be used (badly) for this purpose is just all kinds of wrong.

(*)In brief, A counting semaphore is designed for controlling concurrent access to a resource that is limited but has multiple instantiations.

A queue, even a pre-limited queue, does not fit this criteria.

1. a queue can only be accessed by one user (reader or writer) at a time, regardless of how much is does or can contain.
2. the availability of a queues resources is not determined by the actions of competing consumers -- the reason d'etre of the counting semaphore -- but rather by the actions of the producer.

   They are very different animals.

3. Having the consumer(s) of a queue effectively control the producer(s) for that queue, requires multi-way synchronisation, and defeats the primary purpose of the queue: that of acting as a flexible buffer between producers and consumers to avoid the need for and problems associated with such synchronisation.

   Those problems include:

   Consumers and producers can become 'step-locked'.

   The flow falls into a pattern of the consumer(s) all wasting many time-slices (and expensive context switches) waiting for the producer(s). Then alternately, the producer(s) all waste time-slices waiting for the consumer(s). The flow becomes lock-stepped wasting many context switches between each forward step.

   The very purpose of queues is to alleviate this problem by acting as a flexible buffer removing the need for synchronisation.

See 940030 for a little more discussion.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Thanks, that is a good solution, though as you say, not very robust.

I think a robust implementation would be a good idea, so with that in mind I have emailed the author of Thread::Semaphore about adding the feature, either via a subclass as you suggest, or by adding the feature to the existing core module.

Don't worry, whatever code I submit will robust and be properly tested to avoid the bugs you suggest.