I think you will need multiplexing, so that when multiple connection co-exist, they will all appear to get a timely response. Take a look at IO::Select.

Also, better have timeout defined for your socket, so that you can transfer files, and at the same time, periodically check for new connections through accept(), and accept() will no longer block the process (infinitely).