Well, since it looks like you've already solved your problem, I'll give you a heads up for the thing that cause me some confusion/grief when I started learning sockets.
When you send() data, it won't necessarily all arrive in one recv() call. e.g. If you $server->send('abcde'), you may receive it as 'abc' followed by 'de' in your next loop.
Comment on Re: Timing issues with IO::Socket::INET and IO::Select
Anonymous Monk,
Since socket handles have auto-flush turned on by default using these modules, I assume this is an issue of the client. More specifically, the client doing it's own buffering. This shouldn't be a huge deal since I intend to limit the amount one can say anyway but I am interested in hearing more.