I just added a local $SIG{'PIPE'} = sub { print "got PIPE!\n"; }; handler to the sub with that line. Let's see if that does anything. Since both you and ysth suggest it I'm hopeful that this may indeed be the problem.

Update: Jackpot! It's indeed a SIGPIPE. This also explains why I had trouble creating a smaller sample. There is a timeout on the other side which closes the socket after a certain time of inactivity. As long as at least one of the objects in the queue is making progress there's some communication going and there's no problem. But if all elements in the queue are blocked waiting for a local event the other side will timeout, causing the next attempted write to give the SIGPIPE.

Thanks and ++ to everyone who replied.