If you are actually doing this "while" condition

while (my $client = $server->accept) {
[download]

and the process gets a signal (say SIGCHLD), the accept will return false, with error EINTR (interrupted system call) and your "while" loop will exit.

I've seen this bug in several other's programs and it can cause the intermittent failure you are seeing.

Try something like

while(1) { if( my $client = $server->accept) { #accept code...
[download]