TCP involves protocol handshakes when you setup and tear down a connection. In the case where you're connected to the remote server, the delay might be related to the end points (ie, your machine and the remote server) negotiating the closing of the socket.

Another possible source for the delay could be the buffering that happens automagically on STDOUT. You can get rid of it by saying something like $| = 1; near the beginning of your script. See if this makes the delays go away.