I don't know enough about socket comms on *nix, but can't accept return without a connection? For example, with ECONNABORTED.

As all your server threads are using (cloned copies of) the same server socket backed by the same physical resource, is it possible that if one client aborts its connection, that could cause all your threads to receive that error from the accept, causing their while loops to terminate?