Look at IO::Select if your connect times are short and fast. I would guess that you can add filehandles from different ports to the same select object.

If select don't work for you, like for big file transfers, you will need a forking or threaded server. See Glib based forking server with root messaging for example, or google for "perl forking server" or "perl threaded server".

Otherwise, I havn't a clue what you want. :-)