I reproduced the time difference on Mac OS X (real 1.120s vs. 0.381s).

The time difference seems to disappear when I change the server code to read:

...
            my $pid = fork;
            if ( ! defined $pid ) { die "Can't fork: $!" }
            if ( $pid ) {
                close $sock;
            }
            else {
                while( 1 ){
...
[download]

(Each "..." is code I didn't change. The salient changes are (1) it will die when it can't fork, and (2) the parent will close the socket it won't be using anyway.)

I'm not sure why having the parent leave the socket open causes the problem you're having, but it doesn't surprise me that it causes some problem.

Also, you should be checking (in the client and in the server) that your sockets are created successfully. In some runs, but not all, I got a message that implies to me that the call to IO::Socket::INET->new() failed.