This should be pretty much impossible and I at least am
unable to reproduce it. Do you have some way to trigger this so we can see the effect too ? On which OS and version is this ?
If you use the list-form of accept, what is the value of
the peer information you get from that for the case the
getpeername(2) calls fail ? Can you get an strace/truss from
around when this happens ?