The problem is that the SIGCHLD is interrupting the accept() call. To fix:

use Errno qw(EINTR);

...
while (1) {
 $client = $socket->accept;
 if (!$client) {
   next if $! == EINTR;
   die "Error on accept: $!\n";
 }
[download]

or equivalent

Updated per request:
http://archive.develooper.com/perl5-porters@perl.org/msg85068.html
http://www.gnu.org/manual/glibc-2.2.5/html_node/Interrupted-Primitives.html

This is also one of the examples given in Worse is Better