Without seeing the server code, I would guess you don't have the Reuse or Listen options set to the proper values.

my $server = IO::Socket::INET->new
   (
    LocalAddr    => '192.168.0.1',
    LocalPort    => 1337,
    Type         => SOCK_STREAM,
    Reuse        => 1,
    Listen       => 10
    ) or die "could not open port\n";

warn "server ready waiting for connections.....  \n";
[download]