Problem might be on the Server, error message makes me think it is, but this may benefit there as well, setting ReuseAddr to 1.

my $socket = IO::Socket::INET->new(
   ReuseAddr => 1,
[download]

Socket options are documented in socket;7 man 7 socket

Your server could be blocking on a <$fh> call if that doesn't work.