Oh, sorry for the misunderstanding ...
Are you passing a timeout to the IO::Socket constructor?
Also, are you forking after the creation of $sock? if so, all the child processes call accept() upon $sock?
A .t file that triggers the "bug" would be more than welcome.