The way you have chosen to do it so far looks a bit convoluted.
Check out IO::Socket::INET - reimplementing it as a simple pre-forking TCP server could be the answer.
Remember to set ReusePort / ReuseAddr while testing :)