You just include the listening socket in your select() or IO::Select and only call accept on it when there is activity on it (can_read if you are using IO::Select). That way you won't block.

Updated: If you are using Perl 5.8, be aware that any socket call might return EINTR, which just means you should call it again. This wasn't true in earlier versions of Perl and is not covered in most examples/documentation. See 244415 for a little more info.