You are probably not setting the 'reuse' flag with your listening server. Whenever a network socket is bound to a port, used, and then closed, that port is still "reserved" and cannot be used again for a small period of time. This is to keep any residual packets that might still arrive for that application to hit and be discarded, instead of making their way into the next application that happens to use that port erroneously.

If you're using IO::Socket (which you really should be using), set the 'Reuse' argument to a true value, otherwise you'll have to call setsockopt directly:

setsockopt(SERVER, SOL_SOCKET, SO_REUSEADDR, pack("l", 1));
[download]

See your system documentation for 'setsockopt' for more information about valid arguments. See perlipc for additional information about using network sockets in Perl.