Add ReuseAddr => 1 to your param list. The OS is trying to help you by not letting any lingering packets in a connection accidently go to a new process. If you are sure that this will never happen (or don't care), you can tell the kernel that you'll live dangerously by permitting quick re-use of the same port number.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.