It's been a while since I've done much work at this layer but it looks as though you are running a single threaded server listener and so you must wait for the TIMEOUT release by the server. This is an important necessary requirement - you do not want another client to connect to your TCP connection and you do not want to lose the connection if it is interrupted for a minute.
I may not be explaining this very well but is worth reading more on single and multi-threaded TCP servers.
A good place to start off the top of my search is this SO article. or some of the older PM nodes such as 549414