I spent a few hours and finally found this paramater you can pass to the IO::Socket::INET ... Timeout =>

my $sock = new IO::Socket::INET (
         LocalPort => '1401',
         Proto => 'tcp',
         Listen => 1,
         Reuse => 1,
         Timeout => .1,
);
[download]

After .1 seconds it times out the accept command and continues on through your loop happily. As an added benefit this helps you prevent race conditions.