Try it this way:

#! perl -slw
use strict;
use IO::Socket;
use threads qw[ yield ];
use threads::shared;
use Thread::Queue;

sub worker {
    my $Q = shift;
    while( my $fnum = $Q->dequeue ) {
        open my $client, "+<&$fnum" or die $!;
        while( <$client> ) {
            print $client $_;
            print $_;
        }
        close $client;
    }
}

our $WORKERS ||= 5;

my $Q = new Thread::Queue;
my @workers = map threads->create( \&worker, $Q ), 1 .. $WORKERS;

my $server = IO::Socket::INET->new(
    LocalHost=>'localhost:54321', Listen=>5, Reuse=> 1
) or die $^E;

while( my $client = $server->accept ) {
    my $fno = fileno( $client );
    $Q->enqueue( $fno );
    yield while $Q->pending; ## wait until a worker dups the socket
    close $client;           ## Now we can safely close it.
}
[download]

Rational: The socket won't be released until all open handles to are closed. You are currently getting one handle from the accept; creating a second within the thread (via the fileno). When the client goes away, the threads read loop ends and it's handle gets (explicitly or implicitly) closed. But, the handle in the main thread remains and the socket will not be finalised until it is closed, but there is nothing in the main thread that will do that. If you close the socket prior (or even immediately after!) queuing the fileno for the worker thread, by the time the worker gets it, there is no socket to dup.

So, when accept returns, take the fileno, queue it, then yield your timeslices until the queue is empty. Now, a worker has dequeued the fileno and opened it's own handle to the socket, so it is safe to close the main threads copy. Everyone is happy :)