I hav'nt seen you use IO::Select yet. If you don't want to fork or thread the server's client connections, you could have a 2 step server, where the front end uses IO::Select to just accept connections and push the clients into a FIFO array.
Then you could pop clients off of that array, and run there intense operations in separate threads or forks, by passing the client's socket filehandles off. If it all sounds too complicated for you, try something like
POE, and look thru it's cookbook for server examples.
Also see Simple threaded chat server and look at the server for "multiecho chat" and see how I stuff client sockets into a threads shared array.
You need to think about this stuff for awhile before it makes sense. :-)