The TIME_WAIT state is a protection measure of TCP protocol. If you initiate closure of some TCP connection, then after connection is closed the port used by the connection shall stay open for a while. This is necessary for proper functioning of the TCP.

There's a way to force the system to cancel TIME_WAIT using SO_REUSEADDR, but you are running chance of confusing your own application. While used for server application, this is considered to be of low probability. But you are doing it for client application that connects to the same set of hosts, so you may get unexpected closures from remote server.