Lacking some snipped, I just guess.... After accepting a new connection, your program owns two sockets (and handles to those). One socket still listening for new connections and another one that represents the newly established connection (your telnet session). Maybe you just closed the listening socket instead of that of the new connection by confusing the handles?
Comment on Re: Closing Sockets with IO::Socket::INET