Here's some building blocks for you. It's just an extension of the example at the end of the IO::Select perldoc page with some features from your stated problem added.

#!/usr/bin/perl

# http://perlmonks.org/?node_id=1120916

use IO::Socket;
use IO::Select;
use strict;

my ($port, $current, $cycle) = (shift // 6667);
my $listen = IO::Socket::INET->new(LocalPort => $port, Listen => 9, Re
+use => 1)
  or die "$@ opening socket on port $port";

my $sel = IO::Select->new($listen, 0);

while(1)
  {
  for my $h ($sel->can_read)
    {
    if($h == $listen)
      {
      $sel->add(my $new = $h->accept);
      $current //= $new;
      print "accepted from: ", $new->peerhost, ":", $new->peerport, "\
+n";
      }
    elsif($h == 0)
      {
      sysread STDIN, my $in, 1024; # ignore
      $current = (grep $_ > 0 && $_ != $listen, $sel->handles)
        [++$cycle % ($sel->count - 2 || 1)];
      print $current
        ? "switch to " . $current->peerhost . ':' .  $current->peerpor
+t . "\n"
        : "no clients\n";
      }
    elsif(sysread $h, my $in, 1024)
      {
      $h == $current and print $in;
      }
    else
      {
      $sel->remove($h);
      $h == $current and undef $current, print "client exited\n";
      }
    }
  }
[download]

As others (including me) have said, this kind of problem should be done in one of the async frameworks, but sometimes a plain example can make for a decent tutorial.

Or not.