(everything in this reply is untested)

Your main problem is here:

while (my $conn = $socket->accept) {
[download]

Accept failing is not usually fatal and should not terminate your loop. In particular, if you get a signal, accept can return without result with $! set to EINTR. And you get signals when your children die.

This doesn't happen in somewhat older perls since they by default have a setting to restart slow systemcalls (but the problem would reappear in really old perls which didn't yet do that).

In new perls (since 5.8) signals only set a flag and handlers only get executed if the perl dispatcher is in a safe state and sees the flag. But that forced the developers to turn off the restarting of slow systemcalls since otherwise actual running of the signal handler code can be indefinitely delayed (the flag would get set, but since the systemcall gets restarted it still doesn't get to the dispatcher, so nothing gets executed)

On UNIX versions that support it, you could avoid getting child signals at all by using

$SIG{CHLD} = "IGNORE";
[download]

But that just sets you up to take the fall when you start handling other kinds of signals.

So just change the loop to something like:

use POSIX qw(EINTR);
...
while (1) {
   my $conn = $socket->accept;
   if (!$conn) {
       warn("Accept error: $!" if $! != EINTR;
       next;
   }
   ....
}
[download]