Thanks for the response.

The server is using SO_REUSEADRR (I think?) and an extended queue

use IO::Socket::INET;
my $server = IO::Socket::INET->new(
    LocalPort => 9000,
    Listen   => 1000,
    Reuse    => 1.
) or die "Create listner failed with: $! [$^E]";
[download]

but the confusing thing is that I am seeing no errors from the server, only the clients. For testing I am using this client:

#! perl -slw
use strict;
use List::Util qw[ shuffle ];
use IO::Socket::INET;;

our $N ||= 10000;

my $cno = shift;

for ( 1.. $N ) {
    warn $_;
#    <STDIN>;
    my $c = IO::Socket::INET->new( 'localhost:9000' ) or do{
        warn "\a\n\n\t\t*** $cno ($_): Connect failed with [$^E] ***\n
+\n\a";
        sleep 10;
        next;
    };
    $c->shutdown( 2 ) or warn "Shutdown failed with $^E\n"
}
[download]

And starting multiple copies like this

c:\test>for /l %i in (1,1,10) do @start /b serialClient.pl %i
[download]

If I uncomment the <STDIN>; line, and start 10 (or 20 or 50) copies, and hold down the enter key, I get anywhere from 3,500 to 10,000 connect/shutdown cycles by each client, cleanly before the errors start to occur (in the clients). When the errors occur, all activity in the server ceases for a while, 5 to 15 seconds. If I wait until the server starts servicing (queued) connects again and then hold the return key down, everything is fine again and all the clients are able to complete the connect/shutdown cycle again as fast as the machine allows them to run.

Sometimes this happy state continues till the end of the run; sometimes the error condition reoccurs, with identical symptomsm, and can again be cured by pausing the clients until the server starts to show activity again.

I also have a version of the client that conducts a conversation with the server, passing a url and retrieving the content from it as gathered by the server (being served from diskcache to alleviate the narrowness of my connection). When the clients start being unable to get a new connection--those already connected continue to retrieve their data without errors.

When the clients are unable to connect, the server shows no unusual activity. It is multithreaded with the actual conversations be handed off to a pool of threads once the accept is returned. If the clients simply disconnect, the thread pool remains basically inactive and the main thread with the accept loop doesn't appear to be shuffling memory or anything.

I've also tried disconnecting from the internet and terminating my firewall to exclude that from the equation with no change.

It's like the server just decides to stop accepting requests for a while until thing quieten down. I'm beginning to suspect that the problem is deeper than Perl and lies somewhere in the protocal stack?